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1 1th  Annual  Systems  Engineering  Conference 

San  Diego,  CA 

20-23  October  2008 


Agenda 


TUESDAY.  21  OCTOBER  2008 


Keynote  Addresses: 

•  HON  Charles  McQueary,  Director,  Operational  Test  &  Evaluation; 

Plenary  Session:  Executive  Panel 

Moderator: 

Ms.  Kristin  Baldwin,  Deputy  Director,  Software  Engineering  &  System  Assurance 
Panelists: 

•  Mr.  Terry  Jaggers,  Director,  SAF/AQR  (Science,  Technology  &  Engineering) 

•  Mr.  Carl  Siel,  Chief  Systems  Engineer;  ASN(RDA)CHENG 

•  Mr.  Ross  Guckert,  Assistant  Deputy,  Acquisition  &  Systems  Integration  ASA(ALT) 
Luncheon  with  Speaker  in  the  Regatta  Pavilion 

•  Dr.  Ronald  Jost,  Deputy  Assistant  Secretary  of  Defense,  C3,  Space  &  Spectrum 

BAYVIEW  III:  SYSTEMS  ENGINEERING  EFFECTIVENESS 
Session  2C1 


o  7099-  DoD’s  Systems  and  Engineering  Revitalization  Efforts-  An  Update  Mr.  Nicholas  M.  Torelli,  OSD/SSE/ED 
o  7475  -  The  Effectiveness  of  Systems  Engineering  on  Federal  (DoD)  System  Development  Programs  -  Update  2008,  Mr.  Ken  Ptack 
o  7153-  Systems  Engineering  Plan  (SEP)  and  Systems  Engineering  Management  Plan  (SEMP)  Unification  Mr.  Chet  Bracuto,  OSD 
°  Naval  Power  21  Integration  &  Interoperability  Improvement,  Mr.  Kevin  Smith 
o  7089  -  Systems  Engineering  for  Systems  of  Systems,  Dr.  Judith  Dahmann,  The  MITRE  Corporation 

BAYVIEW  II:  TEST  &  EVALUATION  IN  SYSTEMS  ENGINEERING 
Session  2C2 

o  7100-  Implementation  of  the  2007  Developmental  Test  &  Evaluation  Defense  Science  Board  Results:  Mr.  Chris  DiPetto,  OUSD/SSR/ 
o  7101  -  Test  and  Evaluation  Value  Metrics  at  Acquisition  Decision  Points:  Ms.  Darlene  Mosser-Kerner,  OUSD/SSE/DTE 
°  6979  -  Integration  of  Software  Intensive  Systems:  Mr.  Tom  Wissink,  Lockheed  Martin 
o  6996  -  Modeling  &  Simulation  in  the  Test  &  Evaluation  Master  Plan,  Mr.  Michael  Truelove 
o  7 1 03  -  “New. . .  .Improved”  Test  &  Evaluation  Master  Plan,  Ms.  Darlene  Mosser-Kerner 
o  7290  -  Mission  Based  T&E  Strategy,  Mr.  Chris  Wilcox 

BAYVIEW  I:  PROGRAM  MANAGEMENT 
Session  2C3 

°  7096  -  New  Acquisition  Policy  and  Its  Impact  on  Defense  Systems  Engineering:  Ms.  Sharon  Vannucci,  ODUSD/SSE/ED 
o  6919-  Improving  the  Quality  of  DoD  Weapon  Systems:  Ms.  Cheryl  K.  Andrew,  U.S.  Government  Accountability  Office 
o  An  Air  Force  S&T  Directorate’s  View  on  Applying  Systems  Engineering  Principles  to  its  Programs 

o  High  Confidence  Technology  Transition  Planning  Through  the  Use  of  Stage-Gates  (TD-13),  Dr.  Claudia  Kropas-Hughes,  HQ,  AFMC 
°  7002  -  Systems  Engineering  Re-vitalization  at  the  Defense  Contract  Management  Agency:  Mr.  Lawrence  F.  Cianciolo,  Defense  Contract  Management 
Agency 

MISSION  I  SYSTEM  SAFETY-  ESOH  &  HSI 
Session  2C4 
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o  6997  -  Human  Systems  Integration  and  Model  Based  Systems  Engineering:  Dr.  Abraham  W.  Meilich,  Lockheed  Martin 

o  7084  -  Human  Reliability  Analysis  and  the  Advanced  Man  Portable  Air  Defense  System:  A  Case  Study:  Mr.  Christopher  A.  Brown,  Naval  Surface 
Warfare  Center,  Crane 

o  7092-  Systems  Engineering  to  Ensure  Aircraft  Airworthiness,  Mr.  Jim  Miller 
o  7161  -  ESOH  In  Acquisition  OSD  Expectations  For  Implementing  DODI  5000.02,  Ms.  Karen  Gill 
©  ESOH  Challenges  in  Commissioning  an  Aircraft  Carrier,  Mr.  Doug  Parrish,  Booz  Allen  Hamilton 

MISSION  II  MODELING  &  SIMULATION 
Session  2C5 

o  7172  -  Execution  of  the  Acquisition  M&S  Master  Plan-  A  Progress  Report:  Mr.  James  W..  Hollenbach,  Simulation  Strategies,  Inc. 
o  Update  on  Survey  on  Modeling  and  Simulation  Support  for  the  Systems  Engineering  of  Systems  of  Systems,  Ms.  Judith  Dahmann,  Simulation 
Strategies,  Inc 

o  7440  -  Synchronizing  Modeling  and  Simulation  Plans  Across  Navy  Acquisition:  Dr.  Ivar  Oswalt,  VisiTech 
o  7085  -  Modeling  and  Simulation  Resource  Reuse  Business  Model:  Mr.  Dennis  P.  Shea,  Center  For  Naval  Analyses 

o  Joint  Rapid  Scenario  Generation  (JRSG)  System  Engineering,  Mr.  Ralph  O’Connell,  US  Joint  Forces  Command,  Joint  Capability  Development  (J8) 
o  Cross-Command  Collaboration  Effort  (3CE) 

MISSION  III:  NET  CENTRIC  OPERATIONS 
Session  2C6 

o  7461 -Network  Centric  Engineering  use  of  the  NCOIC  (Network  Centric  Operations  Industry  Consortium)  Processes  and  Tools  in  a  Logistics  Example: 
Mr.  Thomas  M.  Dlugolecki,  SenseResponder  LLC 

o  7128  -  Changing  the  Value  Equation  in  Engineering  and  Acquisition  to  Align  Systems  of  Systems  with  Dynamic  Mission  Needs:  Mr.  Philip  J.  Boxer, 
Software  Engineering  Institute 

°  7341  -  Crucial  Factors  in  the  Design  of  Net-Centric  Systems:  Dr.  David  Hernandez,  Tactronics  Holdings,  LLC 
o  7330  -  Creating  a  Systems  Architecture  for  an  SOA-based  IT  System  as  Part  of  a  Systems  Engineering  Process,  Mr.  Robert  S.  Elinger 
o  A  Service-Oriented  Architecture  (SOA)  Business  Model  for  the  U.S.  Department  of  Defense  (DoD) 

PALM  I:  REQUIREMENTS  DEVELOPMENT  &  MANAGEMENT 
Session  2C7 

o  7444-  Acquisitions  Requirements  of  Capabilities  in  a  Netcentric  Enterprise  -  Creating  a  Capabilities  Engineering  Framework:  Mr.Jack  M.  Van  Kirk, 
SFAE-AV-AS 

o  7138-  Implications  of  Capability-based  Planning  on  Requirements  Engineering:  Mr.  Leonard  Sadauskas,  DoD  CIO,  IT  Investment  &  Commercial 
Policy 

o  7191-  System  Concept  of  Operations:  Standards,  Practices,  and  Reality:  Ms.  Nicole  Roberts,  L-3  Communications 
o  7066  -  Two-Step  Methodology  to  Reduce  Software  System  Requirements  Defects,  Mr.  Robert  J.  Kosman 
°  7451  -  Why  Design  for  Testability  Sooner?,  Mr.  Bruce  Bardell,  BAE  Systems 

o  7399  -  The  Challenges  of  Requirements  Decomposition,  Ms.  Eliza  Siu,  Northrop  Grumman  Corporation 

PALM  II:  SOFTWARE 
Session  2C8 
Panel 

o  7137  -  DoD  Software  Engineering  and  System  Assurance:  Moderator:  Ms.  Kristen  J.  Baldwin,  Systems  and  Software  Engineering 
°  7139  -  A  Framework  for  Integrating  Systems  and  Software  Engineering:  Dr.  Richard  Turner,  Stevens  Institute  of  Technology 
°  7041  -  Software  Process  Improvement  for  Acquisition  of  Naval  Software  Intensive  Systems:  Mr.  Carl  Siel,  U.S.  Navy,  ASN  (RDA)  CHENG 
o  71 19  -  Architecting  Systems  to  Meet  Expectations  -  Managing  Quality  Characteristics  To  Reduce  Risk,  Mr.  Paul  R.  Croll,  CSC 
o  7156  -  New  Concepts  and  Trends  -  How  Future  Trends  in  Systems  and  Software  Technology  Bode  Well  for  Enabling  Improved  Acquisition  and 
Performance  in  Defense  Systems,  Dr.  Kenneth  E.  Nidiffer 

°  7239  -  Systems  and  Software  Design  Principles  for  Large-Scale  Mission-Critical  Embedded  Products  from  Aerospace  and  Financial  Problems 
Domains,  Mr.  Rick  Selby,  Northrop  Grumman  Space  Technology 

WEDNESDAY.  22  OCTOBER  2008 


Luncheon  with  Speaker  in  the  Regatta  Pavilion 

°  Ms.  Shannon  Cunniff,  Director,  Emerging  Containments:  Office  of  Under  Secretary  of  Defense  (Installations  and  Environment) 

BAYVIEW  III:  SYSTEMS  ENGINEERING  EFFECTIVENESS 
Session  3A1 

o  7405  -  Systems  Engineering:  Application  in  Complex  Organizations:  Mr.  Kevin  Roney,  Booz  Allen  Hamilton 
o  7065  -  Establishing  a  Systems  Engineering  Center  of  Excellence  in  PEO  Ground  Combat  Systems:  Mr.  Michael  H.  Phillips,  Jacobs 
o  7423-  Systems  Engineering  Capability  Development:  Mr.  Edward  Andres,  TARDEC 

Session  3B1 

°  7436-  A  Process  Decision  Table  for  Integrated  Systems  and  Software  Engineering:  Dr.  Barry  Boehm,  USC-CSSE 
o  7190  -  A  Tool  to  Enhance  Systems  Engineering  Planning:  Ms.  Sue  O’Brien,  The  University  of  Alabama  in  Huntsville 
o  6945-  The  Role  of  Chaos  and  Complexity  in  Systems  Development:  Dr.  Robert  J.  Monson,  Lockheed  Martin 

Session  3C1 
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°  6878  -  Reduction  of  Total  Ownership  Costs  (R-TOC)  and  Value  Engineering  (VE)  in  Defense  System’s  Life  Cycle:  Mr.  Chet  Bracuto,  OSD 
o  7007  -  Using  Performance-Based  Earned  Value(R)  for  Measuring  Systems  Engineering  Effectiveness:  Dr.  Ronald  S.  Carson,  Boeing 
o  7017-KBAD-  A  Cost-Effective  Way  to  Conduct  Design  and  Analysis:  Dr.  Steven  Dam,  Systems  and  Proposal  Engineering  Company 
o  6886  -  Air  Force  Systems  Engineering  Assessment  Model,  Mr.  Randy  Bullard 
o  7030  -  Defining  100  Best  Practices  for  SE,  Mr.  Ian  Talbot,  AAC/EN 

o  7204  -  Advancing  Systems  Engineering  Practice  within  the  Department  of  Defense:  Overview  of  DoD’s  Newest  University  Affiliated  Research  Center 
(UARC),  Ms.  Sharon  Vannucci,  ODUSD 
o  7093  -  Systems  Engineering  Performance  Measures,  Mr.  Jim  Miller 

BAYVIEW  II:  TEST  &  EVALUATION  IN  SYSTEMS  ENGINEERING 
Session  3A2 

o  6937  -  Systems  Engineering  for  Testing  in  a  Joint  Mission  Environment:  Mr.  Earl  Reyes,  OSD/JTEM 
°  7209-  Joint  Mission  Environment  Test  Capability  (JMETC):  Mr.  Chip  Ferguson,  JMETC 
o  735 1  -  End  to  End  System  Test  Architecture:  Dr.  Masuma  Ahmed,  Lockheed  Martin 

Session  3B2 

o  701 1  -  Implementing  a  Methodology  to  Incorporate  Operational  Realism  in  CONOPS  &  Testing:  Mr.  William  R.  Lyders,  ASSETT,  Inc. 
o  6928  -  The  Role  of  T&E  in  the  Requirements  Process  for  System  of  Systems:  Mr.  Walter  C.  Reel,  Naval  Surface  Warfare  Center  -  Dahlgren 
o  7372  -  Integrated  T&E  Process  and  Tools  in  the  Joint  High  Speed  Vessel  Program:  Mr.  Stephen  F.  Randolph,  Alion  Science  and  Technology 

BAYVIEW  II:  BEST  PRACTICES  &  STANDARDIZATION 
Session  3C2 

o  6874  -  Why  CMMI  Isn’t  Enough:  Ms.  Anita  Carleton,  Software  Engineering  Institute 
o  6888  -  Value  Engineering:  Enhance  DMSMS  Solutions:  Dr.  Jay  Mandelbaum,  Institute  for  Defense  Analysis 

o  7761-  Applying  Business  Process  Modeling  to  Develop  Systems  Engineering  Guidance  for  New  DoD  Acquisition  Regulations:  Dr.  Judith  Dahmann, 
OSD 

Session  3D2 

°  7003  -  How  to  Specify  Applicable  Documents:  Mr.  James  R.  van  Gaasbeek,  Northrop  Grumman 

o  7014  -  Systems  Engineering  in  the  Science  and  Technology  Environment  -  Best  Practices  and  other  Lessons  Learned  from  the  Air  Force  Research 
Laboratory:  Mr.  William  P.  Doyle,  General  Dynamics 

o  7031 -Lessons  Learned  Doing  Systems  Engineering  Assessments  on  the  Government:  Mr.  Ian  Talbot,  AAC/EN 

BAYVIEW  I:  PROGRAM  MANAGEMENT 
Session  3A3 

°  7438  -  The  Incremental  Commitment  Model  and  Competitive  Prototyping:  Dr.  Barry  Boehm,  USC 

o  7070  -  An  Integrated,  Knowledge-based  Approach  to  Developing  Weapon  System  Business  Cases  could  Improve  Acquisition  Outcomes:  Mr.  Travis  J. 
Masters,  U.S.  Government  Accountability  Office 

o  7258  -  Joint  Service  Safety  Testing  Study  Phase  II  Final  Presentation,  Ms.  Paige  V.  Ripani,  Booz  Allen  Hamilton 

Session  3B3 

o  7340  -  “Integrated  Management  Operating  Model  (iMOM)”,  An  E-2D  Advanced  Hawkeye  SD&D  Program  Case  Study:  Mr.  Douglas  J.  Shaffer, 
Northrop  Grumman 

o  7269-  Closing  the  Gap  Between  Systems  Engineering  and  Project  Management:  Mr.  Robert  W.  Ferguson,  Software  Engineering  Institute 
o  7349-  The  Death  of  Rish  Management:  Mr.  Michael  P.  Gaydar,  Naval  Air  Systems  Command 

Session  3C3 

o  7095  -  Evaluating  Complex  System  Development  Maturity-  The  Creation  and  Implementation  of  a  System  Readiness  Level  for  Defense  Acquisition 
Programs:  Mr.  Eric  Forbes,  Northrop  Grumman 

o  7023-  Program  Management  of  Concurrently  Developed  Complex  Systems  -  Lessons  Learned:  Mr.  Alexander  Polack,  The  Aerospace  Corporation 

Session  3D3 

°  7385  -  Enabling  More  Effective  Weapons  Systems  Acquisition  and  Sustainment  through  an  Enterprise  Approach:  Mr.  John  Stewart,  Oracle 
o  7462  -  Applying  the  Tenets  of  Military  Planning  and  Execution  to  Project  and  Systems  Engineering  Management:  Mr.  Philip  Lindeman,  SAIC 
o  7479  -  360  Degree  View  of  the  Technology,  Strategy  and  Business:  Mr.  Min-Gu  Lee,  Lockheed  Martin 


MISSION  I:  SYSTEM  SAFETY-  ESOH  &  HSI 
Session  3B4 

o  721 1  -  Defining  a  Generic  Hazard  Tracking  Database  for  Future  Programs:  Mr.  Jeff  Walker,  Booz  Allen  Hamilton 
o  7215  -  DoD  Energy  Demand:  Addressing  the  Unintended  Consequences:  Mr.  Thomas  Morehouse,  Booz  Allen  Hamilton 
°  7258  -  Joint  Service  Safety  Testing  Study:  Ms.  Paige  Ripani,  Booz  Allen  Hamilton 

Session  3C4 
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o  Update  on  Revisions  to  MIL-STD  882:  Mr.  Robert  “Bob”  Smith,  Booz  Allen  Hamilton 


MISSION  II:  MODELING  &  SIMULATION 
Session  3A5 

o  7347  -  Deployment  of  SysML  in  Tools  and  Architectures:  an  Industry  Perspective:  Mr.  Rick  Steiner,  Raytheon 

°  7073  -  Standardized  Documentation  for  Verification,  Validation,  and  Accreditation  —  An  Update  to  the  Systems  Engineering  Community:  Mr.  Kevin 
Charlow,  Space  and  Warfare  Systems  Center-Charleston 

o  7052  -  Architecture  and  Model  Based  Systems  Engineering  for  Lean  Results:  Mr.  Tim  Olson,  Lean  Solutions  Institute,  Inc. 

Session  3B5 

o  7026  -  Rapid  Assessment  Approach  Using  Commander’s  Intent  to  Identify  Promising  Force  Structure  Architectures  for  System  Trade  Studies:  Mr. 
David  A.  Blancett,  Northrup  Grumman 

°  7082  -  Domain  Modeling:  A  Roadmap  to  Convergence:  Mr.  Nathaniel  C.  Homer,  The  Johns  Hopkins  University  Applied  Physics  Laboratory 
©  7364  -  Predictive  Modeling:  Principles  and  Practice:  Dr.  Rick  Hefner,  Northrop  Gmmman 

Session  3C5 

o  7144  -  Systems  Engineering  Analysis  of  Threat  Reduction  Systems  using  a  Collaborative  Constmctive  Simulation  Environment:  Dr.  James  E. 

Coolahan,  Johns  Hopkins  University  Applied  Physics  Laboratory 
o  7393  -  Systems  Engineering  Approach  to  Total  Vehicle  Design  and  Integration:  Mr.  Walter  J.  Budd,  BAE  Systems 

Session  3D5 

o  7228  -  Total  System  Modeling:  A  System  Engineering  Application  of  the  Higraph  Formalism:  Mr.  Kevin  Fogarty,  SAIC 
©  7077  -  Near-field  RCS  and  Fuze  Modeling  and  Simulation:  Mr.  David  Hall,  Survice  Engineering  Company 
o  7174  -  Virtual  Battlespace  Center  for  Systems  Engineering:  Mr.  James  Hollenbach,  Simulation  Strategies,  Inc. 

MISSION  III:  NET  CENTRIC  OPERATIONS 
Session  3A6 

o  6954  -  SOAs  and  Net-Centric  Warfare-Similarities,  Differences  and  Conflicts:  Mr.  James  A.  Mazzei,  The  Aerospace  Corporation 
°  7374  -  Capitalizing  in  Migrating  Web  Service  Environments:  Mr.  Brian  Eleazer,  South  Carolina  Research  Authority 

Session  3B6 

o  6972  -  A  System  Engineering  Approach  to  Develop  a  Service-Oriented  Perspective:  Mr.  Rob  Byrd,  SI  International 
o  7413  -  Systems  Engineering  Approach  for  Assessing  a  Warfighter’s  Cognitive  Performance:  Mr.  James  Buxton,  U.S.  Army 

Session  3C6 

o  7105  -  Building  Net-Ready  Information  Interoperability  Performance  Indicator  Widgets  For  DoDAF  2.0  Dashboards:  Mr.  William  B.  Anderson, 
Software  Engineering  Institute 

°  7088  -  The  Benefit  of  Collaboration:  Integration  between  the  DoDAF  and  Systems  Engineering  Communities:  Mr.  Tim  Tritsch,  Vitech  Corporation 
o  7337  -  Modeling  Cognition  in  the  DoD  Architecture  Framework  for  Early  Concept  Development:  Dr.  John  M.  Colombi,  Air  Force  Institute  of 
Technology 

o  7046  -  Survivable  Network  Design  Framework,  Mr.  Dennis  Moen,  Lockheed  Martin 

o  7377  -  Joint  Surface  Warfare  Joint  Capability  Technology  Demonstration  -  Maturing  Weapon  Data  Link  Concepts  into  Operational  Capability,  Mr. 
Robert  Finlay  son,  John  Hopkins  University 

PALM  I:  REQUIREMENTS  DEVELOPMENT  &  MANAGEMENT 
Session  3A7 

o  7047-Stop  the  Pain:  Take  Some  Requirements  Definition  and  Management  for  Project  Success:  Mr.  Scott  Derby,  AVISTA  Incorporated 
o  7068-Daily  Challenges  in  Requirements  Engineering:  Mr.  Frank  J.  Salvatore,  High  Performance  Technologies,  Inc. 
o  7593-  Correlation  of  Types  of  Requirements  to  Verification  Methods:  Dr.  William  G.  Bail,  The  MITRE  Corporation 

Session  3B7 

°  7548-  Mission  Analysis  and  its  Impact  on  SE  Fundamentals:  Mr.  John  T.  McDonald,  Raytheon 
o  7055-  How  to  Write  ‘Lean  and  Mean’  Requirements:  Mr.  Tim  Olson,  Lean  Solutions  Institute,  Inc. 

PALM  I:  LOGISTRICS,  SUPPORTABILITY  &  SUSTAINMENT 
Session  3C7 

o  7180-A  Continuous  Process  View  of  Systems  Engineering  for  the  Sustainment  Phase:  Mr.  Paul  d.  Ratke,  OC  -  ALC 
o  7183-  Progress  Toward  the  Development  of  a  Reliability  Investment  Cost  Estimating  Relationship:  Mr.  Andy  Long,  LMI 
°  7235-  Future  Combat  Systems  (FCS)  Logistics  Systems:  Ms.  Soo  R.  Yoon,  Boeing 

Session  3D7 

°  7390  -  Systems  Engineering  of  Deployed  Systems:  Mr.  Robert  K.  Finlayson,  Johns  Hopkins  University,  Applied  Physics  Laboratory 
o  7383  -  Extending  Enterprise  Systems  for  an  Integrated  Logistics  Management  Environment:  Mr.  Mike  Korzenowski,  General  Dynamics  Land  Systems 
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7455-  The  Seven  Affordability  Sins  of  Logistics  System  Integration:  Dr.  Thomas  E.  Herald,  Lockheed  Martin 

PALM  II:  SOFTWARE 
Session  3A8 

o  7114-  Building  the  Next  Generation  of  Software  Engineers  -  Benchmarking  Graduate  Education:  Dr.  Arthur  Pyster,  Stevens  Institute  of  Technology 
o  7135  -  Improving  Work  Breakdown  Structure  (WBS)  Guidance  for  Weapons  Systems  with  Substantial  Software  Content:  Mr.  Christopher  Miller, 
OUSD/SE/SSA 

o  7232  -  ASN  (RD&A)  Initiatives  to  Improve  Integration  of  Software  Engineering  into  Defense  Acquisition  Related  Systems  Engineering:  Dr.  John  F. 
Miller,  The  MITRE  Corporation 

Session  3B8 

o  7198-  Software  Reuse  Readiness  Levels:  A  Framework  for  Decision  Making:  Mr.  Steven  Wong,  Northrop  Grumman 
o  7195  -  Counting  Software  Size:  Is  it  as  easy  as  Busying  a  Gallon  of  Gas?:  Ms.  Lori  Vaughan.  Northrop  Grumman 

PAM  II:  ARCHITECTURE 
Session  3C8 

°  7136-  Architecture  Trade-off  Analysis  Method®  (AT AM®)  for  System  Architecture  Evaluation:  Mr.  Michael  Gagliardi,  Software  Engineering  Institute 
o  7243  -  Method  for  Aligning  Architecture  Frameworks  and  System  Requirements:  Mr.  Richard  L.  Eilers,  IBM 

Session  3D8 

■  7428-  Adaptable  Architecture  for  System  of  Systems:  Mr.  Bruce  Schneider,  Applied  Physics  Lab  Johns  Hopkins  University 

■  7285  -  Universal  Architecture  Description  Framework:  Mr.  Jeffrey  O.  Grady,  JOG  System  Engineering 

■  7109  -  Applying  Open  Architecture  Concepts  to  Mission  and  Ship  Systems:  Mr.  John  M.  Green,  Naval  Postgraduate  School 

■  7273  -  US  Air  Force  Global  Persistent  Attack  Architecture,  Process,  &  Risk  Analysis:  Maj  Jeffrey  D.  Havlicek,  Air  Force  Center  for  Systems 
Engineering 


THURSDAY.  23  OCTOBER  2008 


BAYVIEW  III:  SYSTEMS  ENGINEERING  EFFECTIVENESS 
Session  4A1 

■  7697  -  Enhancing  Systems  Engineering  in  the  Department  of  Defense:  Mr.  Ceasar  Sharper,  ODUSD  /SSE 

■  7186  -  Air  Force  Implementation  of  NRC  “Pre-A  SE”  Study  Committee  Recommendations:  Mr.  Jeff  Loren,  AF/AQRE 

■  7281 -A  Holistic  Approach  to  System  Development:  Mr.  Douglas  T.  Wong,  NASA  Johnson  Space  Center 

Session  4B1 

■  7004  -  Operational  Concepts:  Mr.  James  R.  van  Gaasbeek,  Northrop  Grumman 

■  7296  -  The  Dangers  of  Oversimplifying  Availability:  Dr.  Jeffrey  M.  Harris,  General  Dynamics 

■  7214-Developing  and  Maintaining  the  Technical  Baseline:  Mr.  Michael  G.  Ucchino,  Air  Force  Institute  of  Technology 

Session  4C1 

■  7289  -  Process  Tailoring  Patterns  and  Frameworks  for  Accelerating  Systems  Engineering  Processes:  Mr.  Larry  J.  Earnest,  Northrop  Grumman 

■  7054  -  Using  Lean  Principles  and  Process  Models  to  Achieve  Measurable  Results:  Mr.  Tim  Olson,  Lean  Solutions  Institute,  Inc. 

■  7265-  Rocket  Motor  Development  Cycle  Time  -  Business  Process  Review:  Mr.  Jose  Gonzalez,  OUSD/PSA/LW&M 

BAYVIEW  II:  BEST  PRACTICES  &  STANDARDIZATION 
Session  4A2 

■  7076  -  Systems  and  Software  Life  Cycle  Process  Standards:  Foundation  for  Integrated  Systems  and  Software  Engineering:  Ms.  Teresa  Doran, 
TECHSOFT 

■  71 1 1  -  Improving  Process  Utilizations  with  Tools:  Mr.  Frank  J.  Salvatore,  High  Performance  Technologies,  Inc. 

■  7179  -  Integration  of  Systems  and  Software  Engineering:  Implications  from  Standards  and  Models  Applied  to  DoDs’  Acquisition  Programs:  Mr. 
Donald  Gantzer,  ODUSD/SSE 

Session  4B2 

■  7325  -  Applying  CMMI  High  Maturity  Practices  and  Leveraging  LEAN  Six  Sigma:  Mrs.  Ann  Hennon,  BAE  Systems 

■  7422  -  NDIA  CMMI  Working  Group:  Status  and  Plans:  Mr.  Geoff  Draper,  Harris  Corporation 

■  7441  -  Process  Enrichment  Boot  Camp,  Mr.  Victor  Elias,  High  Performance  Technology,  Inc 

■  7446  -  Best  Practices  Clearinghouse:  Making  Lessons  Learned  Come  Alive  and  Be  Practical,  Mr.  Forrest  Shull,  Fraunhofer  Center,  Maryland 

MISSION  II:  EDUCATION  &  TRAINING 
Session  4A5 

■  6944  -  Establishing  the  Need  for  Functional  Analysis  in  Systems  Development:  Dr.  Robert  J.  Monson,  Lockheed  Martin 

■  6946  -  Improving  Systems  Engineering  Execution  and  Knowledge  Management:  Mr.  Steven  C.  Head,  Boeing 

Session  4B5 


1  lth  Annual  Systems  Engineering  Conference.html[5/25/2016  9:20:56  AM] 


Untitled  Document 


■  7094  -  Development  and  Validation  of  a  Systems  Engineering  Competency  Model:  Dr.  Don  Gelosh,  SAIC 

■  7098  -  Accelerate  Performance  Improvements:  Systems  Engineering  Skills  Competency  Analysis  and  Training  Program  Development:  Mr. 

Steven  A.  Diebold,  General  Dynamics, 

■  7130  -  Concept  Defmiti-  A  Historical  Perspective:  Dr.  David  R.  Jacques,  Air  Force  Institute  of  Technology 

MISSION  III:  ENTERPRISE  HEALTH  MANAGEMENT 
Session  4A6 

■  7580  -  Engineering  Solutions  for  Fleet  Readiness  Centers  utilizing  an  Avionics  Rapid  Action  Team  Innovation  Cell:  Mr.  Bill  Birurakis,  PIDESO 

■  7447  -  Prognostics  as  an  Approach  to  Improve  Mission  Readiness  and  Availability:  Mr.  Sony  Mathew,  Center  for  Advance  Life  Cycle 
Engineering 

■  7613  -  Prognostics  Based  Health  Assessment  System  Approaches:  Mr.  Ronald  D.  Newman,  VSE  Corporation 

Session  4B6 

■  7520  -  NDIA  ID  Electronic  Prognostics  (E-Prog)  Task  Follow-on  Study  to  Quantify  Weapon  System  Benefits:  Mr.  Paul  Howard,  Paul  L.  Howard 
Enterprises 

■  7597  -  Enterprise  Health  Management  Emerging  Technology  Transition  Enabling  Plan:  Mr.  Chris  H.  Reisig,  Boeing 

LRU  Prognostics  Demonstration  Video  MPEG  Video  RealPlayer 

PALM  I:  LOGISTICS,  SUPPORTABILITY  &  SUSTAINMENT 
Session  4A7 

■  7481-  Defining  the  Prognostics  Health  Management  Enterprise  Architecture:  Mr.  Ethan  Xu,  Raytheon 

■  7131-  Sustaining  Systems  Engineering  -  The  A- 10  Example:  Dr.  David  R.  Jacques,  Air  Force  Institute  of  Technology 

■  7188-  Reliability  Centered  Maintenance  Applied  to  the  CH-47  Chinook  Helicopter-Universal  Principles  that  go  beyond  Equipment  Maintenance: 
Ms.  Nancy  Regan,  The  Force,  Inc. 

Session  4B7 

■  7207-  Sustainment  Engineering  versus  Systems  Engineering,  Is  There  A  Difference?:  Ms.  Karen  B.  Bausman,  AF  Center  for  Systems 
Engineering 

■  7064-  Reliability  Growth  Analysis  of  Mobile  Gun  System  during  PVT:  Dr.  Dmitry  Tananko,  GDLS 

PALM  II:  ARCHITECTURE 
Session  4A8 

■  7401-  Enabling  Systems  Engineering  with  an  Integrated  Approach  to  Knowledge  Discovery  and  Architecture  Framework:  Mr.  Michael  R. 
Collins,  Advantage  Development,  Inc. 

■  7453  -  Open  Architecture  in  Electronics  Systems:  Mr.  Bruce  R.  Bardell,  BAE  Systems 

■  7069  -  The  Value  of  Architecture:  Mr.  Frank  J.  Salvatore,  High  Performance  Technology,  Inc. 

Session  4B8 

■  7365  -  Enabling  the  Successful  Transition  from  Architecture  to  Concept  Design:  Mr.  Chris  Ryder,  Johns  Hopkins  University  Applied 
Physics  Laboratory 

■  7079  -  The  Benefits  of  Synergizing  Naval  Open  Architecture  Practices  and  Principles  with  Systems  Engineering  Processes:  Mr.  Mike 
Dettman,  PEO  C4I  -  NAVSEA 

■  7029  -  Concurrent  Increment  Sequencing  and  Synchronization  with  Design  Structure  Matrices  in  Software-Intensive  System  Development: 
Dr.  Peter  Hantos,  The  Aerospace  Corporation 
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CONFERENCE  OBJECTIVES 

This  conference  seeks  to  create  an  interactive  forum  for  Program  Managers,  Systems 
Engineers,  Software  Engineers,  Chief  Scientists,  and  Engineers  and  Managers  from 
government,  industry,  and  the  academic  communities  whose  interests  converge  on 
Defense  acquisition,  from  capabilities  analysis  through  operations  and  disposal.  This 
conference  will  provide  the  opportunity  to  learn  from  ones  peers  on  latest  techniques 
and  methodologies,  and  help  shape  policy  and  guidance  through  the  exchange  of 
innovative  procedures  and  lessons  learned  to  address  the  following  current  issues: 

•Effectiveness  of  Systems  Engineering 
•Program  Management 
•Architectures 

•Requirements  Development  &  Management 
•Interoperability  &  Systems  Integration 
•Software  &  Software-intensive  Systems 
•Network  Centric  Operations 
•System-of-Systems  Engineering 
•Modeling  &  Simulation 
•Integrated  Risk  Management 
•Aging  Aircraft 

•Logistics  &  Supportability  including  Performance  Based  Logistics 
•Life  Cycle  Systems  Management 

•Improved  Cycle  Times  for  Design,  Manufacture,  &  Repair  Process 
•Sustainment  &  Upgrade  of  Legacy  Systems 

•Application  of  Government  &  Industry  “Best  Practices”  Tools,  Methodologies,  & 
Technologies 

•System  Safety  -  Environment,  Safety  &  Occupational  Health  &  Human  Systems 
Integration 

•Improved  Mission  Readiness  &  Systems  Availability 
•Enterprise  Health  management  &  Integrated  Diagnostics 
•Systems  Engineering  Training  &  Education 
•Capability  Maturity  Model  Integration  (CMMI) 

•Integrated  Systems  Engineering,  Test,  &  Supportability  Discipline 
•Application  of  DoD  Initiatives: 

-Performance  Based  Business  Environment 
-System  Safety 
-Open  Systems 

-Simulation  Based  Acquisition 
-COTS  Integration 


BACKGROUND 

The  Department  of  Defense 
has  been  undertaking  a  major 
transformation  of  our  military  capability 

over  the  past  few  years  in  response  to  the  new  world  environment 
and  unforeseen,  ever-changing  threats.  The  ability  to  effect  this 
transformation  can  only  be  realized  if  our  Defense  Systems — space, 
air,  land,  sea,  and  under  sea — can  effectively  satisfy  mission  area 
and  capability  requirements,  and  achieve  and  sustain  a  high  degree 
of  interoperability,  systems  integration,  readiness,  availability,  and 
systems  safety,  with  affordable  cost.  We  believe  that  the  greatest 
opportunity  to  achieve  these  objectives  for  new  and  legacy  systems 
is  through  strong  technical  management  embodied  in  systems 


engineering  methodologies  and  processes,  on  the  part  of 
both  industry  and  the  DoD,  in  not  only  the  technical 
arms  but  the  management  &  program  management 
arms.  Strong  emphasis  on  systems  engineering  across  the  full 
acquisition  life  cycle,  from  concept  development  &  refinement 
through  deployment  &  sustainment,  is  a  key  enabler  of  improved 
performance  in  the  overall  acquisition  process  and  effectiveness. 
The  Systems  Engineering  Conference  is  an  annual  event  targeted  at 
exploring  the  role  of  technical  planning  and  execution  in  Defense 
programs  and  systems  from  a  variety  of  perspectives,  academic 
and  pragmatic,  by  the  entire  Defense  systems  engineering 
community. 
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GENERAL  INFORMATION 


CONFERENCE  ATTIRE 

Appropriate  dress  for  this  conference  is  business  casual  for  civilians  and  class  B 
uniform  for  military. 

During  conference  registration  and  check-in,  each  participant  will  be  issued  an 
identification  badge.  Please  be  prepared  to  present  a  picture  ID.  Badges  must  be 
worn  at  all  conference  functions. 


CONFERENCE  PROCEEDINGS 

Proceedings  will  be  available  on  the  web  through  the  Defense  Technical  Information 
Center  (DTIC),  and  will  be  available  one  to  two  weeks  after  the  conference.  You 
will  receive  notification  via  e-mail  once  proceedings  are  posted  and  available  on  the 
web. 


OTHER  INFORMATION 

Conference  Chair:  Mr.  Bob  Rassa,  Raytheon 

Conference  Technical  Program  Co-Chairs:  Dr.  Thomas  Christian,  USAF, 
Technical  Advisor,  Systems  Engineering,  USAF  AFMC/ASC;  Mr.  Steve  Henry, 
Northrop  Grumman 

Plenary:  Ms.  Kristen  Baldwin,  OSD/SSE 

Systems  Engineering  Effectiveness:  Mr.  A1  Brown,  Boeing;  Ms.  Sharon  Vannucci,  OSD 

Logistics  Supportability  &  Sustainment:  Mr.  Joel  Moorvich,  Raytheon 

Involving  Test  &  Evaluation  in  SE:  John  Lohse,  Raytheon;  Darlene  Mosser-Kerner,  OSD 

Program  Management:  Mr.  Hal  Wilson,  Northrop  Grumman 

Modeling  &  Simulation:  Mr.  Jim  Hollenbach,  SIMSTRAT,  Inc.;  Mr.  Gary  Belie, 
Lockheed  Martin 

Net  Centric  Operations:  Mr.  Jack  Zavin,  ASD(NII);  Dr.  Rich  Eilers,  IBM 
Best  Practices  &  Standardization:  To  be  announced 
Software:  Mr.  Paul  Croll,  CSC 

Education  &  Training  in  SE:  Mr.  Mike  Ucchino,  USAF/AFIT/CSE 

Enterprise  Health  Management:  Mr.  Dennis  Hecht,  Boeing;  Mr.  Howard  Savage, 
Savage  Consulting 

System  Safety,  ESOH  &  HIS:  Mr.  Sherman  Forbes,  USAF;  Ms.  Paige  Ripani, 
Booz  Allen  Hamilton 

Requirements  Development  &  Management:  Mr.  Bob  Scheurer,  Boeing 
Architecture:  Mr.  Joe  Kuncel,  Northrop  Grumman;  Mr.  John  Palmer,  Boeing 
Practical  SE  Experience:  To  be  Announced 
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CONFERENCE  AGENDA 


SUNDAY,  OCTOBER  19, 2008 

5:00  pm  -  7:00  pm 

Registration  for  Tutorials  and  General  Conference 
(Tutorials  are  an  additional  $250.00  registration  fee) 

MONDAY,  OCTOBER  20, 2008 

7:00  am  -  5:00  pm 

7:00  am  -  8:00  am 

Registration 

Continental  Breakfast  for  Tutorial  Attendees  ONLY 

8:00  am  -  12:00  pm 

(Tutorials  are  an  additional  $250.00  registration  fee) 

Tutorial  Tracks 

(Please  refer  to  the  following  pages  for  Tutorial  Schedule) 

12:00  pm  -  1:00  pm 

1:00  pm  -  5:00  pm 

5:00  pm  -  6:00  pm 

Lunch  for  Tutorial  Attendees  ONLY 

Tutorial  Tracks  Continued 

Reception  in  the  Regency  Annex  (Open  to  All  Participants) 

TUESDAY,  OCTOBER  21, 2008 

7:15  am  -  5:00  pm 

7:15  am  -  8:15  am 

8:15  am  -  8:30  am 

Registration 

Continental  Breakfast 

Introductions  &  Opening  Remarks: 

Mr.  Sam  Campagna,  Directory  Operations,  NDIA\ 

Mr.  Bob  Rassa,  Director,  Systems  Supportability,  Raytheon ;  Chair,  Systems  Engineering  Division 

8:30  am  -  9:45  am 

Keynote  Addresses: 

HON  Charles  McQueary,  Director,  Operational  Test  &  Evaluation; 

Gen  Les  Lyles,  USAF  (Ret) 

9:45  am  -  10:15  am 

10:15  am  -  12:15  pm 

Break 

Plenary  Session:  Executive  Panel 

Moderator: 

Ms.  Kristin  Baldwin,  Deputy  Director,  Software  Engineering  &  System  Assurance 

Panelists: 

12:15  pm  -  1:30  pm 

Mr.  Terry  Jaggers,  Director,  SAF/AQR  (Science,  Technology  &  Engineering) 

Mr.  Carl  Siel,  Chief  Systems  Engineer,  ASN(RDA)  CHENG 

Mr.  Kelly  Miller,  Director,  Systems  Engineering,  NSA 

Mr.  Ross  Guckert,  Assistant  Deputy,  Acquisition  &  Systems  Integration  ASA(ALT) 
Luncheon  with  Speaker  in  the  Regatta  Pavilion 

Dr.  Ronald  Jost,  Deputy  Assistant  Secretary  of  Defense,  C3,  Space  &  Spectrum 

1:30  pm  -  5:15  pm 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

5:15  pm  -  6:30  pm 

Reception  in  the  Regatta  Pavilion 

SYSTEMS  ENGINEERING  CONFERENCE 
PROGRAM  AGENDA 


CONFERENCE  AGENDA,  CONTINUED 

WEDNESDAY,  OCTOBER  22, 2008 

7:00  am  -  5:00  pm  Registration 


7:00  am  -  8:00  am 

8:00  am  -  12:00  pm 

Continental  Breakfast 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

12:00  pm  -  1:30  pm 

Luncheon  with  Speaker  in  the  Regatta  Pavilion 

Ms.  Shannon  Cunniff,  Director,  Emerging  Containments:  Office  of  Under  Secretary  of  Defense 
(Installations  and  Environment) 

1:30  pm  -  5:15  pm 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

THURSDAY,  OCTOBER  23, 2008 

7:00  am  -  3:00  pm 

7:00  am  -  8:00  am 

8:00  am  -  12:00  pm 

Registration 

Continental  Breakfast 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

12:00  pm  -  1:00  pm 

1:00  pm  -  3:00  pm 

Awards  Lunch  in  the  Regatta  Pavilion 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

3:00  pm 

Conference  Adjourns 

Tutorial  Sessions  -  Monday,  October  20, 2008 
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Thursday,  October  23, 2008 


7214-Developing  and  Maintain¬ 
ing  the  Technical  Baseline 

Mr.  Michael  G.  Ucchino,  Air 

Force  Institute  of  Technology 

7422  -  NDIA  CMMI  Working 

Group:  Status  and  Plans 

Mr,  Geoff  Draper,  Harris 

Corporation 

7255-  Integrated  Change 

Control  for  the  Concurrently 

Developed  Complex  Systems 

-  Lessons  Learned 

Mr.  Alexander  J.  Polack,  The 

Aerospace  Corporation 

7278  -  Integrating  Metics  with 

Qualitative  Temporal  Reasoning 

for  Constraint-Based  Expert 

Systems 

Dr.  James  A.  Crowder,  Raytheon 

7130  -  Concept  Definiti-  A 

Historical  Perspective 

Dr.  David  R.  Jacques,  Air  Force 

Institute  of  Technology 

7029  -  Concurrent  Increment 

Sequencing  and  Synchronization 

with  Design  Structure  Matrices 

in  Software-Intensive  System 

Development 

Dr.  Peter  Hantos,  The  Aerospace 

Coporation 

7296  -  The  Dangers  of 
Oversimplifying  Availability 

Dr.  JeJfrey  M.  Harris,  General 
Dynamics 

7400  -  Systems  Engineering 
Initiative  -  How  do  you 
Implement  a  New  Lessons 

Learned  Process  and  Tool  on  a 
Legacy  Program? 

Mr.  Ray  A.  Polo,  Boeing 

74 59  -  Multi-Factor  Risk 
Management 

Ms.  Laura  West,  BAE  Systems 

7102  -  Reengineering  Electronic 

Warefare:  Shifting  From  Platform  - 

To  Capability  -  Centric  Engineering 

Mr.  William  B.  Anderson,  Software 

Engineering  Institute 

7098  -  Accelerate  Performance 

Improvements:  Systems 

Engineering  Skills  Competency 

Analysis  and  Training  Program 

Development 

Mr.  Steven  A.  Diebold,  General 

Dynamics, 

7597  -  Enterprise  Health  Man¬ 

agement  Emerging  Technology 
Transition  Enabling  Plan 

Mr.  Chris  H.  Reisig,  Boeing 

7064-  Reliability  Growth 

Analysis  of  Mobile  Gun  System 

during  PVT 

Dr.  Dmitry  Tananko,  GDLS 

7079  -  The  Benefits  of  Synergizing 

Naval  Open  Architecture  Practices 

and  Principles  with  Systems 

Engineering  Processes 

Mr.  Mike  Dettman,  PEO  C4I 

-  NAVSEA 

7004  -  Operational  Concepts 

Mr.  James  R.  van  Gaasbeek, 
Northrop  Grumman 

7325  -  Applying  CMMI 

High  Maturity  Practices  and 
Leveraging  LEAN  Six  Sigma 

Mrs.  Ann  Hennon,  BAE  Systems 

7363  -  Integrated  Risk  and  Op¬ 
portunity  Management 

Ms.  Audrey  Dorofee,  Software 
Engineering  Institute 

7063  -  Product  Platforms  in 
Support  of  Rapid  Response 
to  DOD  In-Theatre  Force 
Protection  Needs 

Dr.  Steven  B.  Shooter,  Bucknell 
University 

7094  -  Development  and 

Validation  of  a  Systems 

Engineering  Competency  Model 

Dr.  Don  Gelosh,  SAIC 

7520  -  NDIA  ID  Electronic 

Prognostics  (E-Prog)  Task 

Follow-on  Study  to  Quantify 

Weapon  System  Benefits 

Mr.  Paul  Howard,  Paul  L. 

Howard  Enterprises 

7207-  Sustainment  Engineering 

versus  Systems  Engineering,  Is 

There  A  Difference? 

Ms.  Karen  B.  Bausman,  AF 

Center  for  Systems  Engineering 

7365  -  Enabling  the  Successful 

Transition  from  Architecture  to 

Concept  Design 

Mr.  Chris  Ryder,  Johns  Hopkins 

University  Applied  Physics  Laboratory 

Bayview  III 

Systems  Engineering 
Effectiveness 

Session  4B1 

Bayview  II 

Best  Practices  & 
Standardization 

Session  4B2 

Bayview  1 

Program 

Management 

Session  4B3 

Mission  1 

Practical  Systems 
Engineering 

Experience 

Session  4B4 

Mission  II 

Education  &  Training 
Session  4B5 

Mission  III 

Enterprise  Health 
Management 

Session  4B6 

Palm  1 

Logistics,  Support- 
ability  &  Sustainment 
Session  4B7 

Palm  II 

Architecture 

Session  4B8 
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Thursday,  October  23, 2008 


1:30  pm  -  3:00  pm 


Bayview  III 

Systems  Engineering 
Effectiveness 

Session  4C1 

7289  -  Process  Tailoring 

Patterns  and  Frameworks 
for  Accelerating  Systems 
Engineering  Processes 

Mr.  Larry  J.  Earnest,  Northrop 
Grumman 

7 054  -  Using  Lean  Principles 
and  Process  Models  to  Achieve 
Measurable  Results 

Mr.  Tim  Olson,  Lean  Solutions 
Institute,  Inc. 

7265-  Rocket  Motor 

Development  Cycle  Time 
-  Business  Process  Review 

Mr.  Jose  Gonzalez,  O  USD/PSA/ 
LW&M 

Bayview  II 

Best  Practices  & 
Standardization 

Session  4C2 

7441  -  Process  Enrichment  Boot 
Camp  -  An  Intensive  Introduction  to 
a  Generic,  Enterprise-wide,  Strategic 
Communication  and  Continuous 
Improvement  Methodology 

7446-  Making  Lessons 

Learned  Come  Alive  and  be 

Practical 

Mr.  Victor  Elias,  High  Performance 
Technologies  Inc. 

Mr.  Forest  Shull,  Fraunhofer 

Center  Maryland 

Bayview  1 

Program 

Management 

Session  4C3 

7067-  Estimating  Systems 
Engineering  Level  Of  Effort 

Mr.  Frank  Salvatore,  High 
Performance  Technologies,  Inc. 

7189-  The  Integrated  Natural 
Environment  Authoritative 
Representation  Process 
(INEARP)  and  Beyond 

Maj  James  Everitt,  Air  &  Space 
Natural  Environment  M&S 
Executive  Agent 

Mission  1 

Practical  SE 

Experience 

Session  4C4 

7417  -  VIRGINIA 
(SSN-774)  Class 
Systems  Engineering  to 
Reduce  Total  Ownership 
Cost 

7463  -  The  C-17  PIO 
Team 

7497-  Accuracy  Control 
Tools,  Technology, 
and  Processes  used  for 
Addressing  Hull  Fairness 

Mr.  Steve  Lose,  Naval  Sea 
Systems  Command 

Mr.  David  Murray,  Boeing 

Mr.  Stephan  H .  Hankins, 
Northrop  Grumman 

Mission  II 

Education  & 

Training 

Session  4C5 

7308  -  PeaceKeeper 
Intercontinental  Ballistic  Missile 
Systems  Engineering  Case 

Study 

Mr.  Charles  M.  Garland, 

Air  Force  Center  for  Systems 
Engineering 

7474  -  CAPTURE  of  Critical 
Engineering  Skills  and  Knowledge 

Mrs.  Ann  Hennon,  BAE  Systems, 

Promotional  Partner 


Thank  You  to  Our 
Promotional  Partner! 


LOCKHEED 


LOCKHEED  MARTIN  CORPORATION 

Lockheed  Martin  is  a  premier  systems  integrator  and  global  security  enterprise  principally  engaged  in  the  research, 
design,  development,  manufacture,  integration  and  sustainment  of  advanced  technology  systems,  products  and  services. 
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PRODUCT  DEVELOPMENT  - 
ENGINEERING  PERSPECTIVE 


•  Goal:  To  create  a  disciplined 
engineering  framework  which 
supports  customer  focus,  sustained 
innovation,  and  quick  time-to- 
market 
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Tactronics  Divide  and  Conquer 

Failure  is  Not  An  Option  1 


•  The  Two  Components  of  Success: 

-  "Doing  the  right  things"  and  "Doing  things  right" 

-  Focus  and  Execution 


SYSTEM  FUNCTIONAL  TEST 
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Failure  is  Not  An  Option 


Commitment  to  Discipline 


•  Implementing  a  Disciplined  Engineering  Framework  will  initially  make  things 
appear  qualitatively  "slower",  "harder",  "more  bureaucratic",  "less  responsive"... 

•  The  "startup  costs"  associated  with  this  approach  can  often  elicit  significant 
resistance  from  staff  and  management,  however  the  cumulative  effect  is  a  more 
efficient  organization  and  quicker  speed  to  market 
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Failure  is  Not  An  Option ™ 


What  Makes  Engineering  "Net- 
Centric"  Different? 


•  Goal  of  "Net-Centricity":  Get  the  right 
information  to  the  right  decision-makers  at 
the  right  time,  irrespective  of 
physical/organizational  boundaries 

•  Net-Centric  Operations  aim  to  provide: 

-  Shared  situational  awareness  across  the 
battlespace,  resulting  in: 

•  Increased  ability  to  self-synchronize  &  self-task 
resulting  in: 

—  Increased  agility  in  executing  the  mission  and  carrying  out 
"commander's  intent" 
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Failure  is  Not  An  Option ™ 


What  Makes  Engineering  "Net- 
Centric"  Different? 


•  Systems  Engineering  entails: 

-  Defining  desired  customer/stakeholder  capability 

-  Defining  specific  system  requirements 

-  Allocating  those  requirements  to  specific  sub¬ 
systems/software  modules 


^Taclronics 


Failure  is  Not  An  Option ™ 


What  Makes  Engineering  "Net- 
Centric"  Different? 


•  In  the  case  of  Net-Centricity,  the  "sub-systems"  we 
seek  to  integrate  may  already  exist 

•  Consider  the  much-maligned  "stovepipes": 

—  Represent  investment  in  developing 
technologies/platforms  to  carry  out  specific  tasks 
effectively,  sometimes  refined  over  years  of  field 
deployment 

—  Represent  significant  resource  expenditure  in  training 
personnel  to  use  these  tools 

—  Net-Centric  sub-systems  may  be  separated  by  great 
physical  distance,  but  more  importantly,  "virtual  distance" 

—  Technologies  underlying  Net-Centric  capabilities  - 
communications/information  dissemination  -  are 
relatively  dynamic  compared  to  other  technologies 
("internet  pace") 


Qy  Taclronics 


Failure  is  Not  An  Option ™ 


What  Makes  Engineering  "Net- 
Centric"  Different? 


•  In  the  case  of  Net-Centricity,  the  "sub-systems"  we 
seek  to  integrate  may  already  exist 


•  Consider  the  much-maligned  "stovepipes": 


—  Represent  investment  in  developing 
technologies/platforms  to  carry  out  specific  tasks 

-  Levera^^i^fi^^5SS5ilft^re^ne<^  over  Years  °f 


_  Represent  significant  resource  expenditure  in  training 

-  Leverage  existing  personnel  familiarity 


-  Net-Centric  sub-systems  may  be  separated  by  great 

-  Respeetrdifferences^^dagJt  toitiae  liws-sriiefi  need  'virtual  distance" 

-  Technologies  underlying  Net-Centric  capabilities  — 
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What  Makes  Engineering  "Net- 
Centric"  Different? 


•  Approach: 

-  Leverage  components  that  have  been  developed, 
deployed,  and  refined  through  field  testing 

-  Maximally  leverage  knowledge  and  training  that  is 
in  place  to  get  capabilities  into  the  field  quicker 

-  Account  for  differences  across  user  groups,  rather 
than  forcing  adaptation,  by  allowing  for  tailoring 
to  specific  use  cases 

-  Make  systems  extensible  to  incorporate  new 
capabilities 


This  Approach  Applies  Across 
Technology  Areas 


•  Tactronics'  Products  Areas  Where  this  Approach  to  Systems  Engineering  is  Being 
Applied: 

-  Fixed  Computing/Processing 

—  Human-Machine  Interfacing  and  Displays 

-  Mobile  Computing 

-  Navigational/Mapping  and  Sensor  Processing 
—  Networking  Infrastructure 

—  Power  Management 

-  Radio  Management 

-  Specialized  Data  Manipulation/Transport 

•  Audio  Intercommunications 

•  Beyond-Line-of-Sight  Communications 

•  Data  Acquisition/Monitoring  (including  Platform  Telemetry) 

•  Radar  Processing/Display 

•  Video  Processing/Manipulation 
—  Networked/Fixed  Storage  Devices 
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Examp 


i:  "Off-the-Shelf" 
Software 


^97Tasir0Sgs 

Case  Study: 
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Computinq/Displays 
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Case  Study:  Data  Distribution 
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Case  Study:  Radio 
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Management 
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Case  Study:  Power 
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Case 
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Standards-Based  Computing 
&  Networking  Components 


Operation  In  Multiple 
Rugged  Environments 


''Shopping  List"  For  Integrated 
System  Solutions 
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Study:  Systems 
integration 


Platform  Immaterial  Common 
Line  Replaceable  Units  For: 

•  Man  Portable 


•  Vehicular  Platforms 

•  Maritime  Platforms 

•  Rotary  Wing  Aircraft 

•  Fixed  Wing  Aircraft 

•  Forward  Staging  Bases  FSB's 
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ANY  QUESTIONS? 


Contact  Info:  dhemandez@tactronics.com 
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4  Pillars  of  SysML  -  ABS  Example 
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Integrated  Defense  Systems 


1.  Structure 
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Cross  Connecting  Model  Elements 


Raytheon 

Integrated  Defense  Systems 


1.  Structure 
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Key  Considerations  for  SysML  Tool  Raytheon 

Selection  Integrated  Defense  Systems 


■  The  specific  MBSE  method  employed  may  leverage  specific  SysML  features,  but 
may  not  require  other  features.  It  is  appropriate  to  ask  the  following  questions  to 
emphasize  the  features  of  SysML  that  a  successful  tool  deployment  will  need  to 
support. 

-  Which  behavior  representations  are  most  important?  Activity  diagrams?  State  machines? 
Sequence  diagrams? 

-  Will  there  be  a  need  for  item  flow  representation? 

-  What  kind  of  need  will  there  be  for  detailed  performance  analysis  and  parametric 
modeling?  Expression  of  mathematical  equations  relating  parameters  of  system 
elements  may  be  a  very  important  part  of  the  system  development  process/method 
employed. 

-  Will  there  be  a  need  for  algorithm  specification  &  development?  It  may  be  important  to 
express  information  processing  algorithms  explicitly  in  mathematical  form,  using 
constraint  blocks  and  eventually  relating  them  to  specific  blocks  representing  software 
code. 

-  Which  architecting  principles  need  to  be  supported  by  the  tool? 

-  How  will  allocation  be  used?  The  manner  in  which  allocation  is  used  to  guide  the 
development  process  may  dictate  a  set  of  constraints  &  rules  associated  with  allocation 
relationships.  By  enforcing  or  enabling  these  rules,  a  toolset  can  improve  the  efficiency  of 
the  modeling  process. 
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OMG  SysML  Tutorial  (omgsysml.org) 
Water  Distiller  Example 


Raytheon 

Integrated  Defense  Systems 


■  Functional  Analysis  based,  not  OOA 

-  Relies  heavily  on  activity  diagrams  and  functional  allocation 

■  Solution  to  problem  focused  on  activity  modeling,  flow 
allocation,  item  flows  &  parametrics 

-  Heat  balance  of  distiller  relies  on  properties  of  water  flowing  through 
system 

■  Traditional  UML  tools  just  don’t  do  these  things 
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Tool  Comparison 
For  Distiller  Example 


Raytheon 

Integrated  Defense  Systems 


■  No  tool  “fully”  implements  SysML 

■  Clearly,  each  tool  has  strengths  & 
weaknesses 

-  Make  sure  tool  is  compatible  with  your 
method 

■  Other  tools  exist,  but  not  evaluated 

■  RS(X)  is  tool  I’m  least  familiar  with 
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Distiller  Model  Source 
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Distiller  Model  Organization 


Raytheon 

Integrated  Defense  Systems 


Enterprise 

Architect 

Browser 


Project  Browser 
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Rhapsody 
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RS(X) 

Browser 
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Distiller  Value  Types 
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EA  Functional  Allocation 


Raytheon 

Integrated  Defense  Systems 


ad  DistillWder  [Content  •  Svxmlanes 
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Search  Term: 


Search: 

SysML  Allocations 
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Run 


disch  irge: 
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□ 

From  Type 

From  Name 
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To  Name 

Activity 

BoilWater 

allocate 

block 

Boiler 

Activity 

SeparateFlow 

allocate 

block 

TeeFitting 

/ 

Activity 

ControlFlow 

allocate 

block 

Valve  ! 

Allocate  activity  partitions  work  well,  allocation  tables  are  fast  &  easy 
Flow  allocation  not  possible  (object  flow  to  item  flow) 
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Magic  Draw  Functional  Allocation 


Raytheon 

Integrated  Defense  Systems 


ract  Distill  Water (  cold  dirty  :  H2Q,  purs  :  H2Q,  external :  Heat,  discharge  :  Residue  )  [  6.  swimlanes  ]J 


«continuous» 

cold  dirty :  H20 

{stream} 


■^allocate** 

condenser :  Heat  Exchanger 

<=:allocate** 

evaporator :  Boiler 

■^allocate** 

drain :  Valve 

pure :  H2C 

of4 

^a3  :  Condense  SteaQi  ^_^ream} 

^continuous** 

pure : H20 

{stream} 


■^continuous** 

^  discharge :  Residue 

{stream} 


Allocate  activity  partitions  work 
Flow  allocation  works 
Flexible  tabular  view 
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o 

Mr  Eg  Initial  Distiller  Structure  [Dist-ills-, ,, 

@:Q  Distiller  [Distiller:; Distiller  St... 

I-EEI  -condenser  :  C'istllle'- 1 :Pi  . 

:■■■■□]  -drain  :  Distiller:: Distiller  . 

/ 

pO  -evaporator  :  Distiller 
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Raytheon 

Rhapsody  Functional  Allocation  Integrated  Defense  Systems 


r  Action  nodes  do  not  invoke  activities  (no  activity  hierarchy) 
r  No  activity  parameter  nodes  (on  diagram  frame,  or  otherwise) 
r  Action  pin  notation  is  awkward,  pins  not  reused  when  action  referenced 
r  Can’t  distinguish  control  flow  from  object  flow 
r  Tabular  view  &  reports  of  allocation  are  available 
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RS(X)/E+  Functional  Allocation 


Raytheon 

Integrated  Defense  Systems 


®)  Distill  Water 


allocate  ActivitvPai  tition  ■■ 
Partition  :  condenser 


■'allocate  Activity  Partition,  allocat. , 
Partition^  :  evaporator 


■  allocate  ActivityPartition 
Partitions  :  valve 


g 


a 


-■allocated^  pure 

■ill  Condense  Steam 

W  B 

recovered  :  Heat  steam  :  H20 


pure  :  H2Q 


cold  dirty  ;  H20 

i — i — 

cold  dirty  :  H20 

recovered  ;  Heat 

-allocated- 
cl.  Heat  Water 

external  :  Heat 

i _ 


hot  dirty  :  H20 


hot  dirty  :  H2C 


steam  :  H20 

■■allocated''  pre-dischai 
ill!  Boil  Water 

pre-discharge  :  Residue 


external 


:  Heat 


S 


* 


H 

y  < 


« allocated-  discharge  :  Residue 

‘ill  Drain  Residue 

discharge  ;  Residue 

j> 


■  Non-standard  diagram  frame/label 

■  No  unique  action  names  (must  be  same  name  as  activity),  but  allocation  is  unique 

■  Allocation  partitions  work  (automatically  create  allocation  relationships)  to  blocks 
or  parts. 
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EA  ibd/ltemFlow 


Raytheon 

Integrated  Defense  Systems 


ilxl  Distillei  [In rri.il  Distiller  Internal  Block  Diocj ram] 


«blocb 

Distillei 


w  terln 


hep  tin 


«blocb  [ 
H2° 

i — QT 


«BlockProperty» 


cin  ,  ..  r  .  cout 

comlensor  :  Heat  Exchanger 


hout 


hin 

{TF 


cblocks  H20  X 


«BI  ockProperty* 


feedln  _  drainO  Jt 

evaporator : Boiler 


steamOut 


heatln 


«block»  H20 


«block»  H20 


«blocb  Heat 


rui 


«blpcb  Residue 


«BI  ockProperty* 


«blocb  Resid 


'^§1  jdgeOut 


fluidln  .  .  w  .  fluidOut 
dram  : Valve 


vw  terOut 


■  Allocation  works,  but  compartments  not  supported 

■  Can’t  access  value  properties  of  item  properties  (e.g.  temp  of  water  into 
Heat  Exchanger)  ->  can’t  do  parametric  analysis  of  distiller  example. 
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MD  ibd/ltemFlow 


Raytheon 

Integrated  Defense  Systems 


Distiller  [  2  .distiller  block  diagram  (initial  allocated)  ]J 


dirty  water :  H20 
g  in  ■  Fluid 


mainl  :  1-120 
- ►- 


c  out  ■  Fluid 


main2 .  H20 
- ► - 


middle :  Fluid 


condenser 


ca 


^llQcatedFmmr- 
CIA1 
rt  a3 

h  o.ut :  Fluid  h  in  :  Fluid 

— m - 


bottom .  Fluid 


sludgel  :  Residue 
- ► - 


in :  Fluid 


evaporator  :  Boiler 


CH 


allocatedFrom  =  ■:  a2 

lob :  Fluid 


main3 :  H20 

— 4 - 


bottom  Heat 

— m— 


sludge2 :  Residue 

> - 0 


out .  Fluid 


sludge  ■  Residue 


■m 


drain :  Valve 


Q] 


allocatedFrom  =  ©  a4 


main4 :  H20 


ql  :  Heat 
- 


■B 


purified  H20 


q  in  Heat 


■  Diagram  frame  uses  incorrect  nomenclature 

■  Allocation  compartment  incorrect  format 

■  DOES  allow  full  access  to  item  properties 


Rhapsody  ibd/ltemFlow 


Raytheon 

Integrated  Defense  Systems 


ibd  1  Distiller  Structure  -  itemFIOWS  J  «lnternal  Block  Diagram*. 

Distiller  IBD  -  With  Allocations  from  Behavior  to  Structure 


■  Item  flows  and  item  properties  fully  allocable 

-  Item  flows  look  weird,  but  work  fine 

-  ObjectFlows  can’t  be  allocated,  but  ObjectNodes  can. 

■  Full  allocation  compartments  &  callouts 
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RS(X)/E+  ibd/ltemFlow 
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Integrated  Defense  Systems 


©  Distiller 


Connector  1 


dirty  water  :  H2Q 


Connectors 


«itemflow» 


Connectors 


'«itemflow» 


& 


c  out 

t'- 


condenser  :  Heat  Exchanger 


h.ii 


middle 


bottom 


evaporator  :  Boiler 


top 


bottom  1 

—  Is 


Connectors  slud9e  ;  ^idue 


& 


valve  :  Valve 


ql ;  Heat 


q  in  :  Heat 


«itemfloW» 

Connector3 


<<itemflow» 

Connector? 


Connector4 


«itemflow>* 


puriFied  :  H20 

1 


□ 


m 


■  ItemFlows  incorporated  in  RSD  7.0.5/E+  2. 0.5.1,  but 

-  no  icon  or  name/ltemProperty  on  diagram,  ItemFlow  not  associated  with  Connector 

■  Non-standard  diagram  frame/label 

■  Allows  Allocation  of  ObjectFlow  to  ItemProperty,  but  not  to  ItemFlow 

-  no  allocation  compartment/callouts  on  parts 
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MagicDraw  Parametric  Diagram 


Raytheon 

Integrated  Defense  Systems 


class  Distiller  Isobaric  Heat  Balance  [  compsition  of  equations  ]J 


■  Item  properties,  value  types,  units  and  dimensions  fully  supported 
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Rhapsody  Parametric  Diagram 


Raytheon 

Integrated  Defense  Systems 
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EA  &  RS(X)/E+  Parametrics 


Raytheon 

Integrated  Defense  Systems 


O  Isobaric  Heat  Balance 


Rendering  issues  on  cut  &  paste 
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9  |. 
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distiller.main2::H20 


H20  temp  :  °C 


H20  flow  rate  :  gm/sec 


distillei  .main?;  ;H2Q 

i  m~ ) 

6  | 

H20  flow  rate  :  gm/sec 

Connector  4 
distiller.  main4::H20 

H20  flow  rate  :  gm/sec^ 

[F] 


m  rate  :  qm/sec 

* 

H20 1  heat :  cal/gm 


H20  s  heat :  cal/(gm*°C) 
«BindingConnectOi » 

Connector4 


«  BindingConnector » 
Connector5 


m  rate  :  gm/sec 


I  heat :  cal/gm 


«  BindrngConnec  tor  >» 
Connectorl 


dis  tiller,  purified.  H20 1  heat::  cal/gm 


« BindingConnec  tor» 
Connector? 


I  heat :  cal/gm 


q  i  ate  :  cal/sec 


H20  s  heat :  cal/(gm*°CJ 

*  BindingConnec  tor » 
Connectorl 


distillei  .ql::Heat 

0  :  oc 

Qdot :  cal/sec 


H20  I  heat :  cal/gm 


H20  temp  :  K 


H20  I  heat  :  cal/gm 


H20  s  heat :  cal/(gm*°C) 
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EA  &  RS(X)/E+  Parametrics 


Raytheon 

Integrated  Defense  Systems 


■  Both  support  units,  dimensions,  value  types,  constraint 
blocks,  and  parametric  diagrams 

■  Neither  support  value  properties  of  item  properties  on  item 
flows 

-  Item  Flows  incorporated  in  RSD  7.0.5/E+  2.0.5. 1 
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SysML  Diagrams- 
a  Method  for  Model  Integration 


Raytheon 

Integrated  Defense  Systems 


■  3  separate  hierarchies  of  Structure,  Behavior,  and  Data 

-  Usage  (internal  connection)  is  documented  with  separate  diagrams 

■  These  3  hierarchies  maintained  at  Operational  and  System 
level 


Hierarchy 

Usage 

Cross-Connect 

Structure 

bdd 

ibd  ? 

act  (swimlane),  seq  (lifeline,  op) 

Behavior 

bdd 

act,  stm. 

ibd  (itemFlow),  seq  (msgType) 

Data 

bdd 

(none) 

act  (objFlow),  seq  (msg,op),  stm 

bdd  =  Block  Definition  Diagram  (no  DoDAF)  act  =  Activity  Diagram  (OV-5,  SV-4) 

ibd  =  Internal  Block  Diagram  (OV-2,  SV-1,  SV-2)  seq  =  Sequence  Diagram  (OV-6c,  SV-IOc) 

stm  =  State  Machine  Diagram  (OV-6b,  SV-1  Ob) 
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DoDAF  Views  Horizontally  Cross- 
Connecting  a  Complex  SoS  Model 


Raytheon 

Integrated  Defense  Systems 


I.  Operational 

Command,  OpNode 

II.  System 

IIA.  Conceptual 

Multi-Node  System 


MB.  Logical 

Generic  Systems  (C2,  Sensor...) 


IIC.  As-ls 
POR  1 
POR  2 
POR  3 

IID.  To-Be 

Future  System/Standard  1 
Future  System/Standard  2 


Structure 


OV-2 

OV-4 


Behavior 


A 

fov- 


Data 


5’s 


OV-3 


OV-6c’s 


SV-1 

SV-2 


A 

t  SV-4’s 


SV-1 


SV-1  Pc’s  (generic) 


SV-1  Pc’s  (system  1 ) 


SV-6  ibd  (system  2) 


SV-2 


(each  POR) 


ZAs 

f  SV- 


(System  3,  4)) 
4’s  (Sys  3,  4) 


SV-1  Pc’s  (Sys  4) 


(Future  Sys  2) 


A 


(Future  Sys  1) 
SV-4’s 


SV-6  ibd  (Std  1) 


(Sys  3,  4) 


Triangles  represent  hierarchy  diagrams  (no  DoDAF  equivalent) 
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Allocation  Vertically  Cross-Connecting  a 
Complex  SoS  Model 


Raytheon 

Integrated  Defense  Systems 


I.  Operational 

Command,  OpNode 

II.  System 

IIA.  Conceptual 

Multi-Node  System 


Structure 


W 


Behavior 


MB.  Logical  Allocation 
Generic  Systems  (C2,  Sensor...) 


IIC.  As-ls 
POR  1 
POR  2 
POR  3 

IID.  To-Be 

Future  System/Standard  1 
Future  System/Standard  2 


~/m\ 


A)£^ 


Data 


A 


Allocation 


(stdIU  (std2) 


A7V 


Allocation 


Map 


(POR  1, 

2) 


^\(Future  Sys  2)  Future  Sys  1)  (FS  1)^  / 


Triangles  represent  hierarchy  diagrams  (no  DoDAF  equivalent) 
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The  Death  of  Risk  Management 

Michael  Gaydar 

Chief  Systems  Engineer,  NAVAIR 


Self  Destruction 


2008  NDIA  SE  Conference 


N  A  V  A I  R 


Risk  Identification  And  Mitigation  Is 
Required  On  All  Programs, 

However,  Poor  Implementation  And 
Understanding  Of  Risk  Management  Has 
Resulted  In  Unacceptable  Level  Of  Risk 
Assumption. 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 


2 


DOD  RM  Handbook 


NAV^ 


2008  NDIA  SE  Conference 


A  common  misconception,  and  program 
office  practice,  concerning  risk 
management  is  to  identify  and  track 
issues  (vice  risks),  and  then  manage  the 
consequences  (vice  the  root  causes). 
This  practice  tends  to  mask  true  risks, 
and  it  serves  to  track  rather  than  resolve 
or  mitigate  risks. 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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Risk  Defined 


2008  NDIA  SE  Conference 


NAV^ 


DOD  Risk  Management  Guide 


“Risk  is  a  measure  of  future  uncertainties 
in  achieving  program  performance  goals 
and  objectives  within  defined  cost, 
schedule  and  performance  constraints.  ” 


R]3X  JSMdif; 

:i z'a  of  OyarsjgM,  Pinkiro  io  Phm.j  or 


\ 


UijreriJjofJc  P  erfor-m  21  j\ioe  Go-ifo 


Risk  Management 

2008  NDIA  SE  Conference 


•  Risk  Management  Is  Only  A  Subset  Of 
Project  Management 

•  Risk  Identification 

-  Poorly  Understood 

-  Incorrectly  Implemented 

•  Risk  Mitigation  Plans 

-  Inadequate 

-  Outside  Daily  Program  Management 

•  Risk  Realization  Totally  Ignored 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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First  Law  Of  Risk  Management 

2008  NDIA  SE  Conference 


Risk  Management  Programs 
Require  Risky  Programs 


The  of ^vjrDvti»no^  all 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 


6 


Program  Management  By  The  Book 

2008  NDIA  SE  Conference 


•  Requirements  Must  Be  Achievable  And  Documented 

•  Historically  Derived  Basis  Of  Estimate 

•  Integrated  Master  Schedule 

-  All  Tasks  Are  Planned  And  Linked 

-  Well  Constructed  IAW  ANSI  748 

-  Critical  Path  Understood  And  Managed 

-  Fully  Integrated  Supplier  And  Government  Schedule 
Dependencies 

•  Integrated  Data  Environment 

-  Deliverables  Identified  In  Contractual  Language 

-  Deliverables  Integrated  Into  Master  Schedule 

•  Configuration  Management  Established  &  Active 

•  Timely  Problem  Resolution  Across  Contractual  Lines 

•  Alternate  Design  Paths  For  Critical  Technologies 


Risk  Avoidance  Is  The  Goal 


NAV^ 


2008  NDIA  SE  Conference 


Properly  Planned  And  Executed 
Programs  Inherently  Eliminate  And 

Avoid  Risk 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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Second  Law  Of  Risk  Management 


2008  NDIA  SE  Conference 


Trading  Cost-Schedule-Performance 

Is  A  Ponzi  Scheme 


StUHS  Views _ ffiStu  PUgnts- ReservKl  www.sms  com 


fr/nTr 


To  patent  it,  I'd  have  to  understand  it. 
You  may  need  a  different  lawyer. 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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DOD  Handbook  RM  Objective 


NAV^ 


2008  NDIA  SE  Conference 


The  objective  of  a  well-managed  risk 
management  program  is  to  provide  a  repeatable 
process  for  balancing  cost,  schedule,  and 
performance  goals  within  program  funding, 
especially  on  programs  with  designs  that 
approach  or  exceed  the  state-of-the-art  or  have 
tightly  constrained  or  optimistic  cost,  schedule, 
and  performance  goals... 


...Successful  risk  management  depends  on  the 
knowledge  gleaned  from  assessments  of  all 
aspects  of  the  program... 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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Categories  Of  Risk 


N  AV^AI  R 


2008  NDIA  SE  Conference 


Risk 

Technical 

Critical  Design  Elements  Depend  On  Technology  That  Is 
Just  Not  Achievable.  Caused  By  Overreaching 
Performance  Requirements  Embedded  In  KPPs. 

Programmatic 

Resource  Estimates  (Budget  &  Schedule)  Too  Low. 
Caused  By  Insufficient  BOE  Or  Optimism. 

•  Technical  Risk  Against  KPPs  &  Thresholds  Yields  No  Trade  Space 

•  Result:  No  Resource  Increases  Will  Eliminate  Technical  Risk. 

True  Technical  Risk  Will  Always  Result  In  A  Requirements 
Disconnect  When  Realized. 

•  True  Technical  Risk  Requires  Alternate  Design  Paths  That  Deliver 
Lower,  But  Acceptable,  Levels  Of  Performance 

•  Minimum  Acceptable  Performance,  And  Design,  Must  Be 
Achievable  Within  Current  State  Of  Technology. 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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There  Must  Be  Trade  Space 


NAV^ 


2008  NDIA  SE  Conference 


Congjessional 

Domain 

Program  of 
Record 

Risk 

Contingency 

(CAIG)  _ 

Design 

Cost  Estimate 
(Proposals) 


Contractor  & 
Program  Office 
Domain 


Current  EAC 

Threshold 

Requirements 


User  Domain 


Requirements 

Flexibility 


Threshold 
Requirements  Do 
Not  Support  CAIV 
Margin 


CDD 
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Third  Law  Of  Risk  Management 


NAV^ 


2008  NDIA  SE  Conference 


Hope  springs  eternal 

...until  the  spring  dries  up. 
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Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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Ineffective  Mitigation  Paths 


NAV^ 


2008  NDIA  SE  Conference 


•  Technical 

-  Balance  Design  Against  Unproven  Technology 

-  Pursue  Single  Design  Path  Hoping  Testing  Will 
Show  Compliance 

-  Carry  Significant  (RED)  Risk  Beyond  Design 
Closure  (Roughly  PDR) 

•  Execution 

-  Hope  For  Optimistic  Performance  Through 
Management  Challenges 

-  Shift  Risk  To  Suppliers  In  Firm  Fixed  Price  Contracts 

-  Fail  To  Include  All  Aspect  Of  Rebaseline  In  New 
EAC 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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Effective  Risk  Mitigation  Plan 


NAV^ 


2008  NDIA  SE  Conference 


•  Risk  Realization  MUST  Be  Part  Of  Risk  Mitigation 
Strategy 

•  Risk  Mitigation  Steps  Must  Address  Root  Cause 
Uncertainty 

-  Technical:  Demonstrate  Improved  Performance  Predictions  Or 
Alternate  Design  Path 

-  Execution:  Improve  Resource  Estimates 

•  Technical  Performance  Measures  (TPM)  Are  Essential 
To  Mitigating  Technical  Risk 

•  Task  Identification  Is  Essential  to  Mitigating  Execution 
Risk 


Risk  Mitigation  Steps  Should  Not  Be  A  Way  To  Buy  Time 
In  The  Hope  The  Risk  Will  Be  Eliminated 
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Fourth  Law  Of  Risk  Management 

2008  NDIA  SE  Conference 


You  Get  What  You  Pay  For... 

First  Corollary: 

You  Pay  For  Nothing-You  Get  Nothing 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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Risk  Mitigation  Costs 


NAV^ 


2008  NDIA  SE  Conference 


•  Risk  Mitigation  Plans  Are  Unplanned 
Work 

•  Unplanned  Work  Requires  MR  To 
Execute 

•  Risk  Mitigation  Creates  It  Own  Cost  & 
Schedule  Risk 


•  Unfunded  Risk  Mitigation  Is  Unresolved 
Risk 


r 

Risk  Mitigation  Is  A 

“Pay  Me  Now  Or  Pay  Me  Later” 

Decision 

Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 


17 


Summary 

2008  NDIA  SE  Conference 


•  Risks  Are  Rooted  In  Uncertainty 

•  Disciplined  Use  Of  PM  Tools  Is  Required  To 
Identify  Areas  Of  Uncertainty  (True  Risks) 

•  Historical  Execution  And  Standard  Design 
Practices  Normalize  Optimism 

•  Money  And  Time  Doesn’t  Mitigate  All  Technical 
Risk-Requirement  Relief  Only  Solution 

•  Trade  Space  Has  To  Exist 

•  Mitigation  Plans  Must  Attack  Root  Cause  Of 
Risk-Which  Is  Uncertainty 


Michael  Gaydar,  AIR-4.1,  301-757-5549 
Version  5.0 
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End-to-End  System  Test 

Architecture 

Masuma  Ahmed 

Sr.  Manager,  Lockheed  Martin  SSC 

masuma.ahmed@lmco.com 
(408)  742-2553 


LOCKHEED 


10/23/2008 
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Net-Centric  Mission  Operations  Features 


Fully  Synchronized  Interoperable,  Network  of  Networked 
Systems  and  Mission  Capabilities 

*  Networked  Battle  Command  To  The  Warfighter 

*  Networked  Multi -Spectra  I  Air,  Ground,  Space  Sensors  &  Shooters 

*  Rapidly  Reconfigurable  Networked  Real  Time  C4ISR  Capabilities 

*  Adaptable  Information  Formats  for  Command/Mission 

•  Simultaneous  Real-time,  Near-real  Time,  Non-real  Time, 
Applications 

•  Network-Centric  Collaborative  SOA  /  Infrastructure 

•  Seamless  Information  Sharing  Across  Forces,  Multinational 
and  Interagency  Partners 

*  Built-in  Redundancy  with  Operations  Continuity 


Network  of  Networked  C4ISR  Capabilities 


10/23/2008 
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Net-Enabled  Capabilities 


-  Shortens  Chain  of  Attack 

-  Provides 

-  Decision  Superiority 

-  Greater  Speed 

-  Greater  Precision 

-  Capabilities  Supported 

-  Global  Network  Connectivity 

-  Network  Enabled  Platforms/Weapons 

-  Fused  Intelligence 

-  Real  Time  Command  /  Control  &  Situational  Awareness 


Key:  Net-Centric  Operations 
IP-based  Routing,  Shared  Data,  Assured  Service 
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Air  Force  Vision  -  One  Network 


•  Space  Layer 

•  Transformational  Communications  -  Satellite  Constellations,  Operations  & 
Management  Systems 

•  Near  Earth  Orbit  &  Airborne  Layer 

JTRS,  Laser  Optics,  BMC2,  NATO  AGS,  E-2C  Hawkeye,  J-UCAS,  UAVs 

•  Maneuver  Layer  (upper  echelon) 

•  Future  Combat  Systems,  Blue  Force  Tracking 

•  Maneuver  Layer  (lower  echelon) 

•  Sensors,  Weapon  Systems,  Munitions  Data  link 

•  Terrestrial  Layer 

•  GIG-BE,  Teleport,  CAOC 

•  Characteristics 

•  Robust  Self-Forming,  Self-Healing  Network  of  Mobile  War  Fighters 

•  IP  Routing  Platform  For  Information  Flow  Between  Ground,  Air  and  Space 
Networks 


Merged  Defense  and  Space  Infrastructure 
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TEST  &  INTEGRATION  CHALLENGES 


•  Netcentric  Scope  Encompasses  Integration  of  Diverse 

Systems,  Technologies,  Applications,  and  Protocols  Across 
Forces,  Multinational  and  Interagency  Partners.  This  Requires 

•  Understanding,  Test,  and  Verification  of  Transparent 
Interoperability  of  Protocols  and  Systems  Across  Network 
of  Diverse  Networks 

•  Simulating  Communications  Systems,  Sensors,  Weapons, 
and  War  Fighters  in  an  End-to-End  Test  Environment 

•  Data  Consistency  End-to-End 

•  Multi-step  Processing  End-to-End 

•  Assured  Service  Interoperability  End-to-End 


Seamless  Integration  of  Net-Centric  Capabilities  Requires  Robust 

Test  <&  Verification  Environments 
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TECHNOLOGY  &  PROTOCOL  ISSUES 


•  Technology  and  Protocol  Standards  Are  Not  Perfect 

•  Diverse  Systems  Implementing  the  Same  Standards  May  Support 
Different  Requirements  -  Protocol  Interoperability 

•  Testing  and  Verifying  Interoperability  Across  Network  of  Networks  - 
Challenging 

•  OSI  Protocol  Layers  Can  Span  Across  a  Single  or  Many 
Interfaces  Across  Network  of  Networks 

•  Non-transparency  of  Protocol  Layers  -  Costly  Mission  Failure 

•  Isolating  and  Mitigating  Issues  with  E2E  Protocol  Layers  -  Difficult, 
Time  Consuming  and  Costly 

•  Simulating,  Testing,  and  Verifying  Real-time,  Near-real  Time  and 
Non-real  Time  Protocols  E2E  in  Multi-vendor  Environment - 
Complex  and  Time  Consuming 


Test  <&  Verifications  of  Network  of  Networked  Systems 

Logistical  ly  Complex 
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E2E  TEST  CONSIDERATIONS 


•  Any  Net-centric  Mission  Systems  Must  Be 

•  Tested  in  True  Battlefield  Network  Conditions  Prior  to 
Deployment 

•  End  to  End  Protocol  Interoperability  Fundamental  To  Success  of 
Netcentric  Mission  Operations 

•  Network  of  Networked  Systems  Use  Multi-layered  Protocol 
Architecture  to  Communicate  Transparently  Across  Networked 
Systems 

•  Tested  in  a  Distributed  E2E  Test  Architecture  Emulating 
Real-time,  Near-real  Time,  and  Non-real  Time  Protocols, 
Interfaces,  and  Technologies  across  Networked  Systems 

•  Designed  with  Hierarchical  Protocol  Architecture  in  Mind 

•  Emulated  All  Segments 

•  Supporting  Virtual  Test  Systems  For  Multiple  Test  Scenarios 


Net-Centric,  Distributed,  E2E  System  Test  Architecture 
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Test  Architecture  Considerations 


•  Mesh  vs  Hub/Spoke  Architecture 

•  Cost,  Schedule,  &  Protocol  Considerations 

•  Management,  Control  and  Data  Planes 

•  Distributed  vs  Centralized  Control 

•  Overlay  Protocol  Architectures 

•  Security  Architecture,  Protocols  &  Boundaries 

•  Security  at  Physical  and  Higher  Protocol  Layers 

•  Hardware,  Software,  Simulators,  &  Emulators  Integration 

•  Complex  Protocol  Interactions 

•  Architectural  Requirements 

•  Adaptability  to  Changes 

•  Reconfigurable 

•  Remote  Configurability 

•  Multi-  Protocol  Support 

•  Protocol  Fidelity 


Adaptable  A  Reconfigurable  Secured  System  Test  Architecture 
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Distributed  System  Test  Architecture  Issues 


•  Latency 

•  Security 

•  Timing 

•  Data  Integrity 

•  Service  Availability 

•  Race  Conditions 

•  Priorities 


Early  Planning  and  Detail  Requirements 
Specifications  Essential 
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Network  of  Networks  -  A  Simplified  Example 
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Networked  Satcom  Service  Example 


Space 


Special 
Operations 
Force 

Mobile  Base 
Station 


Mobile  Base 
Station 


UAV 


^  Battlefield 
End-Users 


Battlefield 
End-Users  ^ 

Army  Exploit 

Battlefield 

End-Users 


Regional 

Operations  Center 
(ROC) 
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On-net  P2MP  Streaming  Video  - 

On-net  P2P  Data/Voice 
On-net  P2MP  Data/Voice 
Command 

Thick  Lines  -Trunk  Links 
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Networked  Protocol  Layers  (LI,  L2,  L7)  -  Example 


Naval 

Ship 

Network 


LI,  L2 


v  L1.L2 


Access 

Network 


/  LI,  L2 


Airborne 

Network 


LI,  L2 


On-Net 


Testing  L7  Performance  Over  Diverse  LI  /  L2  -  Complex  A 

Challenging 


Masuma  Ahmed 


Protocol  Performance 


•  Physical  and  Data  Link  Layers  (LI,  L2) 

•  Terminate  Between  Adjacent  Systems  In  The  Same  or  Adjacent 
Networks 

•  Data  Link  Layer  Performance  Depends  on  Physical  Layer  Performance 

•  Application  Layer  Transparent  to  Physical  and  Data  Link  Layers 

•  Example:  Physical  Layer  -  RF,  SONET;  Data  Link  Layer  -  Link  16, 
Ethernet  MAC 

*  Application  Layer  (L7) 

•  Traverses  Multiple  Networks  &  Terminates  End  to  End 

•  Rides  On  Diverse  &  Multiple  Physical  &  Data  Link  Layers  (LI,  L2) 

•  Uses  Services  of  Lower  Protocol  Layers 

•  L7  Performance  and  Data  Integrity  Depend  on  Lower  Protocol  Layer 
Performance  (e.g.  Timing) 

•  Example:  Email,  Streaming  Video,  Audio,  File  Transfer,  Web  Browsing 


L7  Performance  Depends  on  LI  /  L2  Performance 
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Distributed  System  Test  Architecture  -  Exampl 


S/C:  Spacecraft  > 

P/L:  Payload  \ 

G/W:  Gateway 
ROC:  Regional  Operations  Center 
NOC:  Network  Management  Center 
SOCC:  Satellite  Operations/Control  Center 
TE:  Terminal 


L1,L2  ' 

Emulation  / 

/ 

/ 

/  / 


c  __ 

S/C  P/L,  G/W 
(Data 
Transport 
Function) 


S/C  Bus,  SOCC 
(Satellite 
Support 
Function) 


Wide  Area 
Network 


L7 


LI,  L2 
Emulation 


LI,  L2 


n  Emulation 


TE,  ROC,  NMC 
(Network 
Access 
Function) 
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Supports  Virtual  Test  Systems  for  Multiple  Test  Scenarios 
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Distributed  System  Test  Architecture  Features 


Emulates  Multi-Segments  -  Space,  Air,  Terrestrial  Systems,  Elements, 
Interfaces,  &  Protocols 

Consists  of  Geographically  Distributed,  Multiple 

•  System  Integration  Labs  (SILs) 

•  Test  Beds 

•  Simulators 

•  Emulators 

•  Control  Centers 

Interconnected  by  Wide  Area  Networks  (WAN) 

Supports 

•  Multi-Element  and  Flight  Element  Integration,  Test  and  Verification 

•  Multiple  Software  and  Database  Integration  and  Integrated  SW  Load  Testing 

•  Prototyping  Hierarchical  Protocol  Layers  and  Interfaces 

*  Simultaneous  Test  and  Verification  of  Command  /  Control,  Application,  Network,  and 
Lower  Protocol  Layers  and  Interfaces 

•  Functions 

•  Performance 

•  Load 

•  Prove  Out  C4ISR  Interoperability  End  to  End 


Supports  Simultaneous  Test  and  Verification 
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Summary 


•  Distributed  Networked  E2E  System  Test  Architecture  is 
Essential 

•  To  Ensure  Interoperability  of  Networked  C4ISR  Capabilities  Across 
Network  of  Diverse  Networks  Before  Deployment 

•  Test  and  Verify  Transparency  of  All  Networked  Protocol  Layers 

•  Emulate  and  Test  True  Battlefield  Conditions  by  Simulating 
Networked  Elements,  Protocols,  Interfaces,  and  Systems 

•  Support  Test-Like-You-Operate 

•  Facilitate  Early  Risk  Reduction 


Verify  Power  of  Networked  War  Fighting  Before  Deployment 
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Predictive  Modeling: 
Principles  and  Practices 

Rick  Hefner,  Dean  Caccavo 
Northrop  Grumman  Corporation 
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Unlimited  Innovation,  Inc. 

NDIA  Systems  Engineering  Conference 
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Background 


NORTHROP  GRUMMAN 


■  Predictive  modeling  relies  on  historical  program  performance 
data  (predictive  analytics)  in  conjunction  with  a  forecasting 
algorithm  model  to  predict  future  outcomes 

-  Ranges  from  simple  extrapolation  techniques  to  sophisticated 
Neural  Network  based  models 

■  This  presentation  will  discuss  the  principles  of  predictive 
modeling,  outline  the  fundamental  methods  and  tools,  and 
present  typical  results  from  applying  these  techniques  to  project 
performance 
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■s  What  is  Predictive  Analysis? 

■  Recent  Trends 

■  Application  to  Program  Performance 

■  Pilot  Results  and  Feedback 

■  Summary 
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What  is  Predictive  Analysis? 


NORTHROP  GRUMMAN 


■  Could  this  network  packet  be  from  a  virus  attack? 

-  Predict  likelihood  of  the  network  packet  pattern 

Anomaly  detection  (outlier  detection) 

-  Similar  questions: 

•  Are  the  hospital  lab  results  normal  (Adverse  drug  effect  detection) 

•  Is  this  credit  transaction  fraudulent?  (fraud  detection) 

■  Will  this  student  go  to  college? 

-  Based  on  Gender,  Parentlncome,  ParentEncouragement,  IQ,  etc. 

-  E.g.,  if  ParentEncouragement=Yes  and  IQ>1 00,  College=Yes 
4  Classification  (prediction) 

-  Similar  questions: 

•  Is  this  a  spam  email?  (spam  filtering) 

•  Recognition  of  hand-written  letters  (pen  recognition) 

■  What  is  the  person’s  age? 

-  Based  on  Hobby,  MaritalStatus,  NumberOfChildren,  Income,  HouseOwnership, 
NumberOfCars,  ... 

-  E.g.,  If  MaritalStatus=Yes,  Age  =  20+4*NumberOfChildren+0.0001*lncome+... 

Regression  (prediction) 
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■  What  is  Predictive  Analysis? 

■S  Recent  Trends 

■  Application  to  Program  Performance 

■  Pilot  Results  and  Feedback 

■  Summary 
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Predictive  Analysis  Trends  -  Adoption  is  on  the  rise 


NORTHROP  GRUMMAN 


■  Predictive  Analysis  is 
becoming  more  prevalent 
and  integrated  in  business 
applications 

o  Example:  Disease 
management  and  evidence 
based  care,  based  on 
historical  diagnosis  and 
procedure  codes  of  patients 

o  Example:  E-Mail  filtering 
using  predictive  analysis 

■  Predictive  Analysis 
algorithms  are  being 
integrated  into  existing 
databases,  data  mining 
tools 

o  Example:  Microsoft  SQL 
Server  2005  has  predictive 
analysis  algorithms 


Example: 

Premium  predictive  analysis  based  filtering  on  e- 
mail,  available  to  any  e-mail  user 
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Predictive  Analysis  Trends  -  Tools  are  becoming  easier  to  use 
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Predictive  Analysis  Trends  -  Model  development  is  more  structured 


Train  the  Model 


Training  Data 


Test  Data 

Test  the  Model 

Prediction  using 
the  Model 


Prediction  Input  Data 


Third  Party 

Off-the-shelf  or  \  Predictive 

[Proprietary  Predictive)  Analysis 

Analysis  Engine  /  tools 


■  Executive  understanding  of  the  creation,  training 

and  testing  of  the  model  is  critical  to  success 

i — i  44 

■  The  Model  gets  more  powerful  and  accurate  as 

i  ir^  i 

the  volume  of  data  fed  into  the  model  increases 

Off-the-shelf  or  proprietary 
Predictive  Analysis  Model 
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Predictive  Analysis  Trends  -  Algorithms  are  available  for  use 
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Data  Mining  Vendors  &  Tools 


■  SAS  (Enterprise  Miner) 

■  IBM  (DB2  Intelligent  Miner) 

■  Oracle  (ODM  option  to  Oracle  lOg) 

■  SPSS  (Clementine) 

■  Insightful  (Insightful  Miner) 

■  KXEN  (Analytic  Framework) 

■  Prudsys  (Discoverer  and  its  family) 

■  Microsoft  (SQL  Server  2005) 

■  Angoss  (KnowledgeServer  and  its  family) 

■  DBMiner  (DBMiner) 

■  Many  others 
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■  What  is  Predictive  Analysis? 

■  Recent  Trends 

■S  Application  to  Program  Performance 

■  Pilot  Results  and  Feedback 

■  Summary 
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Mission  Assurance  Continuum 


NORTHROP  GRUMMAN 


Program  Performance 
Oversight 


Program  Analysis 
Reporting 


Predictive 
Program  Health 


Industry  Minimum 


Industry  Best  Practice 


Industry  Innovators 


Proactive  Program  Management  Reports  based  on  current  and  passed  Predictive  Analysis  based  on 

Program  Portfolio  Management  performance  data  of  portfolio  programs,  Program  Performance  Modeling 

programs,  and  subcontract  reports 


Approach  and  Scope 

•  Self  reported  Program  Portfolio  includes 
critical  and  high  visibility  programs 

•  Standard  Program  Management  Metrics 
collected  on  a  periodic  basis 

Infrastructure  and  Breadth 

•  Program  data  maintained  by  individual 
programs 

•  Summary  information  provided  to 
enterprise  repository 

Data  Requirements 

•  Very  few  metrics  collected  from 
programs 

•  Key  program  metrics  (cost 
performance,  schedule  performance, 
technical  performance,  CPI,  SPI  etc.) 

•  Standardized  program  taxonomy 
information  like  customer,  contract  type 


•  Self  Reported  Program  metrics  collected 
periodically  and  at  specific  program 
milestones 

•  Reporting  analysis  performed  as  needed 


•  Program  data  collected  periodically  into  an 
enterprise-wide  program  management 
repository 

•  Program,  Enterprise  and  Subcontracts 
performance  reports  available 

•  25-100  metrics  collected  from  programs 

•  Key  program  metrics  collected  at  all  specified 
Program  Milestones. 


•  Self  reported  program  metrics,  organizational 
data,  personnel  data  and  customer  reported 
metrics  collected  at  regular  intervals 

•  Predictive  models  developed  using  historical 
data  (leading  indicators  rationalized) 

•  Models  validated  against  historical  data 

•  Holistic  enterprise  wide  approach  to  program 
execution 

•  Models  continually  refined  using  current 
program  performance  data 

•  Sophisticated  predictive  measures  provided  to 
programs  and  enterprise 

•  50-75  metrics  collected  from  programs  and 
refined  to  include  only  the  few  relevant  metrics 

•  Adaptive  approach  to  qualitative  and 
quantitative  performance  indicators 

•  Direct  and  Indirect  metrics  collected  for  the 
programs;  qualitative  information  is  mined 

•  Proactive  responses  based  on  predictive 
analysis  of  ongoing  and  historical  performance 
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Overarching  Objectives  for  Predictive  Modeling 


NORTHROP  GRUMMAN 


■  Provide  program  management  staff  with  Predictive  Models  to 
“test-their-gut”  against  enterprise  experience  data  before 
making  strategic  program  decisions 

■  Develop  Predictive  Models  that  provide  insight  into 
identifying  “headlight  metrics”  that  influence  Schedule  and 
Cost  realism  during  program  execution 

■  Leverage  existing  enterprise  information  to  develop 
Predictive  Models  for  programs 

■  Ensure  that  models  are  extensible  and  automatically 
calibrated  with  additional  data  from  the  program  and 
enterprise 
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Potential  Areas  for  Predictive  Analysis 


NORTHROP  GRUMMAN 


Potential  Predictive 
Analysis  Models  for 
Program  Management 
and  Subcontractor 
Management 

•  Schedule  Risk  at  WBS  level 
based  on  past  performance 

•  Cost  Risk  at  WBS  level 
based  on  past  performance 

•  Technical  Risk  at  WBS  level 
based  on  past  performance 

•  Spending  and  staffing  profile 
for  the  program  life  cycle 

•  Subcontractor  risk  profile 
based  on  past  performance 

•  Sub-tier  quality  at 
subcontract  and  WBS  level 

•  Defect/Aberrations  for  the 
program  life  cycle 

•  Mission  Assurance  models 
based  on  program  category 


Predictive 

Analysis 

Algorithms 

•  Decision  Trees 

•  Naive  Bayesian 

•  Clustering 

•  Sequence 
Clustering 

•  Association 
Rules 

•  Neural  Network 

•  Time  Series 

•  Custom  Model 
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Predictive  Analysis  High  Level  CONOPS 


1)  Enterprise  data  is  mined 
and  analyzed 

2)  Enterprise  models  are 
defined  by  Analysts 

3)  Enterprise  model 
outputs  are  defined  by 
Analysts  and 
customized  by  PM  staff 

4)  PM  staff  use  models 
interactively 

Key  Benefit: 

Leverages  enterprise 
experience  data  and 
sophisticated  algorithms 
into  predictive  models  for 
cost  and  schedule  realism 
checks  during  program 
execution 


© 


Program  Management  Staff  Model  output  review 
'What  If' Analysis 


I 


© 


Predictive  Analysis 


Predictive  Analysis 
Output  Lift  Graph 


Predictive  Analysis 
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The  Predictive  Modeling  Process 


"  Explore  the  Data 

"  Understand  Data 
Relationships 

■  Derive/Enhance  the 
Data 

"  Use  the  Data  to 
Predict 

■  Train  the  Model 


Train 

— 3F — 


Collect  /  Derive 

- - ^ 

i 

Test 
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What  can  be  Predicted  with  Reasonable  Accuracy? 


NORTHROP  GRUMMAN 


Large 
volume  of 
historical 
data 


0 

o 

o 

0 


(/) 

fil 

(D 

0 


Limited 

Historical 

data 


Limited  Number  of 

Programs 

Enterprise 

Experience 

■  Likelihood  or  return  to 

■  Quadrant  2  predictions 

acceptable  performance 

■  Quadrant  3  predictions 

■  Predictive  Program 

■  Early  warning 

Performance 

“headlight  indicators” 

■  Higher  accuracy  based 

on  enterprise 

1 

experience  3 

■  Cost,  schedule  realism 

■  Phase  realism 

■  WBS  Accuracy 

2 

Low  High 


Volume  of  “Like”  Programs 
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Agenda 
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■  Background 

■  Industry  Trends 

■  Application  to  Program  Performance 
■S  Pilot  Results  and  Feedback 

■  Summary 
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Predictive  Modeling  Pilot  Objectives 


NORTHROP  GRUMMAN 


■  Provide  program  management  staff  with  Predictive  Models  to 
“test-their-gut”  against  enterprise  experience  data  before 
making  strategic  program  decisions 

■  Develop  Predictive  Models  that  provide  insight  into 
identifying  “headlight  metrics”  that  influence  Schedule  and 
Cost  realism  during  program  execution 

■  Leverage  existing  enterprise  information  to  develop 
Predictive  Models  for  programs 

■  Ensure  that  models  are  extensible  and  automatically 
calibrated  with  additional  data  from  the  program  and 
enterprise 
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Pilot  Approach 

•  Analyze  and  rationalize  the  available  enterprise  data 

-  Enterprise  Level  Office  of  Cost  Estimation  and  Risk  Assessment  (OCERA) 
data 

-  Division  Level  Stoplight  Program  data 

-  Program  Level  Program  Review  Authority  (PRA)  data  for  relevant 
programs 

•  Develop  predictive  modeling  approach  to  provide  schedule  and  cost 
measures  during  program  execution  phase 

•  Develop  preliminary  predictive  models  using  appropriate  algorithms 
and  mining  existing  enterprise  data 

-  Mining  -  Clustering,  Decision  Trees  and  Naive  Bayesian  Algorithms 

-  Predictions  -  Neural  Network,  Bayesian  Algorithms  and  Clustering 

•  Get  Pilot  participation  from  three  representative  program  types: 

-  Large  Scale  System  Integration  Low  Rate  Initial  Production  program 

-  Medium  Sized  Software  program 

-  Small  IT  System  (Software  and  Hardware)  program 


Key  Benefit:  Leverages  enterprise  experience  data  and  sophisticated 
algorithms  into  predictive  models  for  use  during  program  execution 
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Data  analyzed  for  developing  preliminary  models 


Data 

Stoplight 

OCERA 

PRA 

Data  Period 

2.5  years 

5-6  years 

Past  4  months 

Frequency 

Quarterly/Some 
older  data  is 
monthly 

Major  milestones  or 
annually 

Monthly 

Breadth  and 
depth  of  data 

Monthly  snapshot 
of  key  metrics 

Very  deep,  very 
broad,  with 
significant 
contextual 
information 

Very  deep,  mostly 
snapshot  without 
significant 
contextual 
information 

Approximate 
number  of  data 
elements 

~  20 

~  70  key  attributes 

~40  key  attributes 

Analyzed  enterprise  level  (OCERA),  division  level  (Stoplight)  and  program  level 

(PRA)  data 
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Some  Actual  Data  Types  Used  to  Develop  Predictive  Model  Relationships 


Program  Data 

Contract  Type 

-  CPAF,  FFP,  CPFF 
Type  of  Program 
Period  of  Performance 
Number  of  Milestones 
Number  of  sub-contractors 

-  Subcontract  value 

-  Subcontract  performance 
Total  Value 

Annual  Sales 


Program  Self 
Assessment 

Monthly  Ratings 

-  Schedule 

-  Technical 

-  Cost 

-  Mission  Assurance 

-  Management 

-  Process 

External  Data 


Milestone  Data 

Milestones 

-  Proposal 

-  Contract  Startup 

-  SRR 

-  SDR 

-  Software  Specification 
Review 

-  PDR 

-  CDR 

-  Test  Readiness 
Review 


Number  of  incremental  deliveries 

-  CPARS 

-  Completion 

Average  staff  count 

-  Customer  satisfaction 

SPI,  CPI 

data 

Other  Data 

EAC,  BAC 

-  Award  Fees 

•  Action  Item  Data 

Number  of  EAC  changes 

•  Organization  benchmark 

Number  of  ECR/ECP 

data 

Defects 

•  SLOC,  ESLOC 

-  Injection  by  phase 

•  Productivity 

-  Occurrence  by  phase 

•  Language,  Component 

Skills  Data 

type,  complexity, 

Program  Review  Data 

•  Reuse  ratios 

Project  Initiation  Review  Data 

•  Platform,  environment 

B  Contains  Enterprise,  Division  and  Program  Data 
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Data  Mining  Results 


Column  Name |  5core |  Input 


Bi  l;  ;  ;  ;  ;  ;  ;  ;-;  0.295 

m 

Tech 

::::::::::  0.275 

X 

CustSatis 

vN  ■.■!■:■!■■.!  0.266 

X 

ContractType 

:i±i::i±i±  0.203 

X 

Customer 

tSW  0.154 

X 

Cost 

■  1 

C-J 

■  1 

O 

X 

ContractTypelD 

Pi  0,113 

X 

MA 

0.101 

X 

MonthAndYear 

ITT  0.093 

X 

TypeOfWork 

4-|-|  0.0Q4 

X 

EVM 

*M  0.078 

X 

Organization 

: :  0.075 

X 

EVMReq 

:  :  0.073 

X 

5upl 

■M  0.068 

X 

WorkTypelD 

.  0.062 

X 

UnconPrecon 

0.058 

X 

T2N 

HI  0.034 

Procl 

H!  0.030 

CashFlowDSR 

1  0.023 

POPBegins 

^0.000 

Prediction  Measures 
Schedule 
Cost 


■  The  mining  showed  that  out  of  the  over  125  metrics  and  measures 
some  are  leading  indicators  and  are  more  important  than  others  in 
influencing  cost  and  schedule 

■  While  it  cannot  be  proved  to  be  conclusive  with  the  limited  data  that 
was  used,  the  trends  were  definite 


23 


U  nil  mite  cl - — — - - - - - - 

Innovations 

- inc 


Derivation  of  Data  &  Data  Relationships 


NORTHROP  GRUMMAN 


■  Examples  of  Derived  Data 

-  Number  of  Outstanding  Program  Issues  (with  and  without  recovery  dates) 

-  Variance  in  program  Cost/Schedule/Technical  health  from  month-to-month 

-  Program  Cost/Schedule/Technical  health  trend  from  month-to-month 

-  Variance  in  VAC  from  month-to-month  taken  as  a  percentage  of  the 
current  EAC 

■  Examples  of  Discovered  Relationships 

-  Schedule  Health  is  a  good  indicator  of  program  Overall  Health  recovery 

-  Cost  and  Technical  Health  are  good  indicators  of  program  Overall  Health 
decline 


Better  understanding  of  the  data  allows  for  organization  and 

enhancement  of  the  dataset 
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Model  Development  &  Calibration 


i  *' 


»■ 

ie> 

o- 

□vcfd  Pceuljtkm  % 


•  Modeling  without  applied  domain  knowledge 
or  calibration  resulted  in  lower  accuracy 

•  Association  models  able  to  determine 
relevant  data  attributes 


Model 


□ala  Miami]  Lift  Chari  For  Miami  Ghrucluri;  ffa-Mmnq  Dcclina  Pradichtan 


Calibrated  Model 


•  Incorporating  domain  knowledge  and 
calibration  into  data  mining  resulted  in  higher 
accuracy 

•  Data  relationships  are  more  clearly  defined 


Domain  knowledge  &  calibration  applied  to  data  mining  can 

enhance  the  predictive  model 
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Typical  Results  from  the  Models 


Ability  for  Programs  to  review  the 
predictive  output  from  multiple 
models  to  “test-the-gut”  before 
making  strategic  program  decisions 


#j  COMPASS 

OSJQ 

Pie  Edit  View  Favorites  Tools  Help 

©Back  *  ;  0  g}  K  e 

* 

Address  £| 

NORTHHOR  GRUMMAN 

composs  1 

Portfolio  Assessment  Contract  Analysts  Program  Dashboard  Scorecard  Oc 

cement  library  Repotting  C  entral  t  ofnp*»  Admin  Leg  Out  '  1 

II Shew  All  £5,  <4  Contract:  !  IPISEU  v 

|  Address  |^| 

hJa* 

NORTHROf*  GRUMMAN 

Portfolio  Assessment  Contract  Analysis 

Program  Dashboard 

Scorecard 

Document  Library  Reporting  Central 

gPil 

compass  * 

Compass  Admin  Log  Out 

©  COMPASS 


File  Edit  View  Favorites  Tools  Help 

0Back  ’  [■]  iil  K  ? 
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A 

IPISEU  (02/03/2006) 

L54 

IPISEU  (01/01/2005) 

0.84 

IPISEU  (10/01/2004) 

0.84 
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IPISEU  (04/01/2004) 
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A 
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0.94 
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IPISEU  (07/01/2004) 

0.90 

IPISEU  (04/01/2004) 
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Forecast  Date  01  /01  /2006 

Current 


|  01/10^008  | 


-  (  05AD7/201 0 

Project  End 


Standard  Performance  Metrics 


Metric  /  Strength  of  Prediction  Current 


PA  YTHEON  COMPANY  Sample 
Company 

RA  YTHEON  COMPANY  Sample 
Company 


3.00 

2.00 


3.71 

Alg.:  Neural  Network 

Domain: 427  CRBA  Programs 
Cluster:  Annual  Sales  Var. 
Staff  Count 
Program  Duration 


—77 — 


£  Internet 

Domain:  427  CBRA  Progran 
Cluster:  Annual  Sales  var 
Staff  Count 


■Vi  Generate  Prediction 


FICTIONAL  DATA 
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Typical  Results  from  the  Models 


Ability  for  staff  to  review  status  and 
trends  across  the  portfolio  of 
programs,  across  a  variety  of 
categories 


^COMPASS 


File  Ed*  View  Favorites  loots  Help 
©Back  -  |SJ  <  & 
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Quarter  by  Quarter  Performance  Trends 


Download  to  Excel  |  Printer  Friendly  3  |  Add  to  Favorites 
Search  Crteria: 
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Agenda 
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■  What  is  Predictive  Analysis? 

■  Recent  Trends 

■  Application  to  Program  Performance 
S  Summary 
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Summary  -  Critical  success  factors 


NORTHROP  GRUMMAN 


■  Executive  and  Enterprise  support  and 
understanding  of  long-term  strategic  benefits 

■  Understanding  of  the  types  of  data  and  the 
correlation  between  the  data 

■  Understanding  of  the  various  constituents  in  the 
value  chain  and  the  tools/processes  for  each 
constituent 

■  Prototypes  or  mockups  that  depict  the  results  of 
the  model 

■  Sound  and  robust  technical  architecture 

■  Delivery  mechanism  that  shields  the  complexity  of 
the  model  from  the  end  users 
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More  Information 


■  OLE  DB  for  DM  specification  ■ 

-  http://www.nnicrosoft.conn/downloads/detail 

s.aspx?FamilvlD=01005f92-dba1-4fa4- 

8ba0-af6a1 9d3021 7&DisplavLanq=en 

■  Plug-in 

-  http://www.msnusers.com/AnalvsisService 

sDataMininq/Documents/Files%2FSQL%2 

0Server%20Data%20Mininq%20Pluq%2DI  ■ 

n%20Alqorithms%20%28Beta%202%20% 

2B%2B%29.zip 

-  A  white  paper,  tutorial,  and  complete  ■ 

sample  code  for  Pair-wise  Linear 

Regression 

■  SQL  Server  2005: 

-  www.microsoft.com/sql/2005 

■  Community:  ■ 

-  Microsoft.public.sqlserver.datamininq 

-  Microsoft.private.sqlserver2005.analvsisser 

vices.datamininq  ■ 

-  Groups.msn.com/AnalvsisServicesDataMin 

inq 

■  msdn.microsoft.com  (search  “data  ■ 

mining”) 
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Decision  trees  (classification/regression): 

-  ftp://ftp.research.microsoft.com/users/suraiitc 

/icde99.pdf 

-  http://www.research.microsoft.com/research/ 

pubs/view.aspx?tr  id=81 

-  http://research.microsoft.com/~dmax/publicat 
ions/dmart-final.pdf 

Association  rules: 

-  Apriori  algorithm  (see  Data  Mining  concepts 

and  techniques) 

Clustering 

-  EM:http://www.research.microsoft.com/script 

s/pubs/view.asp?TR  ID=MSR-TR-98-35 

-  K-means  (see  Data  Mining  concepts  and 

techniques) 

Sequence  clustering 

-  ftp://ftp.research.microsoft.com/pub/tr/tr- 

2000-18.pdf 

Time  series: 

-  http://research.microsoft.com/~dmax/publicat 

ions/dmart-final.pdf 

Neural  network 

-  Conjugate  gradient  method  (see  Data  Mining 

concepts  and  techniques) 

Naive  Bayesian 

-  See  Data  Mining  concepts  and  techniques 
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Contact  Information 


Rick  Hefner,  Ph.D. 
Northrop  Grumman  Corporation 
(310)  812-7290 
rick,  hef  ner(a)nqc.  com 
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Effecting  the  Transition  to 
Concept  Design 


Chris  Ryder 

Johns  Hopkins  University 
Applied  Physics  Laboratory 


The  Basic  Question? 


■  What  is  the  Systems  Engineering  community  doing  to 
enhance  the  development  of  systems  our  Warriors  need 
to  execute  their  missions? 

■  Without: 


■  Being  late  to  need 

■  Costing  too  much 

■  Failing  at  the  wrong  time  and  the  wrong  place 

■  Being  too  hard  to: 

■  Operate 

■  Sustain 

■  Does  our  Defense  Acquisition  system  maintain  a  long¬ 
term  focus  on  development  and  acquisition  of  our 
warfighting  systems? 


MIMA 


stwxti  nuujrai  cvucsm  a  muMnativ 


Observations  (by  some  smart  people) 


■  NDIA  Systems  Engineering  Committee 

■  Undersecretary  of  Defense  for  Acquisition, 
Technology  and  Logistics 

■  Department  of  the  Air  Force  Directorate  for 
Science  and  Technology 


UIMI 

unuan . ■UlVIM'tl.lM 
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NDIA  Systems  Engineering  Committee 


■  Issue  Number  ONE: 


Key  Systems  Engineering  practices 
and  procedures  known  to  be  effective 
are  not  consistently  applied  across  all 
phases  of  the  program  life  cycle! 


Why? 


■  “Inconsistent  SE  practices  for  program  planning  and 
execution” 

■  Training  and  Development  of  career  Systems  Engineers 

■  Retirement  of  the  “gray  beards” 

■  Too  busy  doing  the  “day  job”  to  take  the  necessary  time  to 
deal  with  the  basics 

■  Short-term  focus 

■  Programs  working  toward  the  next  big  event 

■  Public  law  on  appropriations  and  contracting 

■  “Will  this  get  me  promoted?” 

■  Bureaucracy 

■  Well-intentioned  policies  hinder  vice  help 

■  Non-technical  bureaucrats  in  key  positions 
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Undersecretary  of  Defense  (AT&L) 

■  The  Honorable  James  Finley  -  Keynote  address  to  the  NDIA 
Systems  Engineering  Conference  (10/23/07) 


Programs  usually  fail  because  they 
are  not  properly  initiated 


MIMA 
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Why? 


■  Requirements  not  well  defined 

■  Requirements  Creep 

■  Inadequate  early  technical  planning 

■  Inadequate  funding  and  schedule  realism 

■  Lack  of  technical  maturity 

■  Insufficient  focus  on  support  and  sustainment 

■  Reliability  the  most  critical  current  problem 

■  The  services  must  pay  this  bill  every  year 

■  Support  and  sustainment  as  critical  elements  of  Total 
System  Effectiveness 

■  Need  for  a  skilled,  clearable  workforce 
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Air  Force  Office  of  Science,  Technology 
and  Engineering 

■  Mr.  Terry  Jaggers  -  address  to  the  NDIA  SE  Conference 
(10/23/07) 


DoD  needs  to  improve  its  ability 
to  perform  Concept  SE! 


jMPklJk 


Why? 


■  What  is  Concept  SE? 

■  Translate  needs  into  a  set  of  requirements  describing 
a  concept  solution 

■  How  does  Concept  SE  relate  to  the  “traditional  life  cycle 
SE  definition”? 

■  Architecture 

■  Engineering  Design 

■  Test  and  Evaluation 

■  Production  and  Deployment 

■  Concept  SE  leads  to  better  military  utility  assessments 
to  evaluate  concept  alternatives 
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Personal  Observations 


■  Misapplication  of  DoDAF 

■  Fundamental  misunderstanding  of  “The  A- 
Word” 

■  Emphasis  of  Product  over  Process 

■  Architecture  views  over  Architecture  model 

■  Viewing  JCIDS  as  a  bureaucratic  control 
mechanism  as  opposed  to  an  engineering 
opportunity 

■  Emphasis  of  the  artifact  over  the  analysis 
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DoDAF  Contributions  to  SE 


Good  Architecture  ->  Effective  Design 


■  A  good  architecture  model  IS  NECESSARY  for  good 
systems  design 

■  Model  traces  back  to  Requirements;  traces  forward  to 
design 

■  Architecture  views  ARE  NOT  limited  to  those  prescribed 
by  DoDAF 

■  DoDAF  presents  the  C4ISR  Viewpoint,  but  is  this 
sufficient? 

■  What  are  the  other  relevant  viewpoints? 

■  Architecture  model  is  fundamental  for  Concept  SE 


UIMI 

unuan  ■UlVIM'tl.lM 


JCIDS  Contributions  to  SE 


Good  SE  Effective  JCIDS 


■  IF  the  engineering  is  done  right  and  the 
analysis  is  thorough,  THEN  the  JCIDS  will  be 
effective 

■  JCIDS  Functional  (Area,  Needs,  Solutions) 
Analyses  are  critical  SE  activities. 

■  Artifacts  will  reflect  the  analysis 

■  This  is  MATERIAL  SOLUTIONS  ANALYSIS 

■  New  DoD  5000  Pre-MS  A 
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Consider  the  Fundamentals  of  SE 


■  Applying  the  “Key  Systems  Engineering  Practices 
known  to  be  effective” 


■  Needs  Analysis 

■  Concept  definition  and  development 

■  Analyses  of  alternatives 

■  Engineering  and  Development 

■  Advanced  development,  system  design  and 
integration 

■  Production  and  Post-deployment  Support 
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Concept  SE  forms  the  foundation  for  system 
development  AND  deployment 


Model-Based  SE 


■  Modeling  is  fundamental  to  Concept  SE 

■  Captures  operational  and  system  requirements 

■  Foundation  for  operational  and  system  architecture 

■  Details  conceptual  and  engineering  design 

■  Facilitates  Software  development 

■  Basis  for  M&S  environment 

■  Details  information  exchanges  and  data  elements 

■  Text  artifacts  (i.e.  specs)  don’t  go  away 

■  Included  in  the  model  as  parameters,  constraints 
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Model  Evolution  and  Relationships 


System 

Hardware 


System 

Software 
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What  is  a  Model? 


■  A  simplified  representation  of  reality 

■  Used  to  mimic  the  appearance  or  behavior  of  a  system 

or  part  (Kossiakoff  &  Sweet) 

■  Abstracts  features  of  situations  relative  to  the  problem 

being  analyzed  (Blanchard  and  Fabriky) 

■  Promote  understanding  of  the  real  system  (underhni) 


If  you  don’t  model  it,  you  won’t  understand  it! 
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Jacobson 


Systems  Engineering  Method 


■  Every  phase  of  the  System  life  cycle  has  some  form  of: 

■  Requirements  Analysis 

■  Functional  Definition 

■  Physical  Definition 

■  Design  Validation 

■  A  more  fundamental  form  of  the  SE  “VEE”,  but  a  little 
more  iterative 

■  Particularly  within  a  given  life  cycle  phase 
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Source:  Kossiakoff  &  Sweet 

—  API 


SE  Method 
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DoD  Product  Life  Cycle  (Simplified) 


A  A 


(Program 

Initiation) 


A 


IOC 


FOC 


Concept 

Technology 

System  Development 

Production  & 

Operations  & 

Refinement 

Development 

&  Demonstration 

Deployment 

Support 

\  Concept 
/  Decision 

/V  Critical 
<  >  Design 

V  Review 

LRIP/IOT&E  A.  Decision 
V  Review 

Pre-  Systems  Acquisition 


ICD  CDD 


Systems  Acquisition 

AX 


CPD 


Sustainment 


ICD  =  Initial  Capability  Document 
CDD  =  Capability  Development  Document 
CPD  =  Capability  Production  Document 
IOC  =  Initial  Operational  Capability 
FOC  =  Full  Operational  Capability 


Proposed  DoD  Life-Cycle 


Up-front  SE  Required  for  JCIDS 
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DoD  Product  Life  Cycle  (Not  Simplified) 


SE  VEE  (Concept  Refinement) 


INPUTS 


*iCD 


*AoA  Plan 
•Exit  Criteria 
•Ak&rrmtfre  Maintenance 
<&  Logistics  Concepts 


OUTPUTS 


/  *Preiitr\  Sys  Spec 
*T&E  Strategy 
+SEP 


"\ 


'Support  &  Maintenance 
Concepts  < fr  Technologies 
* Inputs  to; 

-draft  CDD 
-AoA 

J 


v  -TDS 
\_  -Coat/Manpower  Est. 


inn, 


Develop  Component  Concepts. 
i»e..  Enablin^'Crilical 
Technologies.  Constraint* 

&  Cost/Risk  Drivers 


The  DoD  Product  Life  Cycle 


Integrated  Defense  Acquisition,  Technology,  &  Logistics  Life  Cycle  Management  Framework 

“  -y.-r  vViie  m  4ki.tr  mj-  »  ft m y.1  n-r  iH-*  : — .--f  1 1  rl  Mr  f-~ ' t:  wu'iw  ■  t*Ji  k  nwf  <.-y  Tw{J t-t mi ■ 


Planning, 
Programming. 
Budgeting, 
ft  Execution 
Process 
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Pre-MS  A  JCIDS  Functional  Analyses 


Joint 

Capabilities 
Integration  & 
Development 
System 

(need  driven) 


Functional 
Area  Analysis 

I 


-  Decision  Points/Milestones 

Strategic  Guidance 

_  f  Z. _ 

F  Li  mi  ly  of  Joint  Future  Concepts 
Concepts  of  Op  erations 
Joint  Tasks 


WL 


Functional 

Nee-ds 

Analysis 

is 


JCD 


- Integrated 


Architectures  ^ . 


Ideas  tor 
Non-materiel 
Approaches 
idotmlpf 

AnafrBfcs-i 


Ideas  for 
Materiel  j- 
Approaches 


Fun  a  ion  Hi  Solution  1 
Analysis 


Analysis  of 
Materiel 
Non-Maleriel 
Approaches 

■  In  i 


Approach  1  | 
Approach  2 1 

\  Appi-oact^H  | 


£&mc&/JROCi 


Post 

Independent 

Analysis 


I'JJL'i-.'THbi-.W  ■ 
^pr?iz.-hfls  : 


There  is  no  “VEE”  during  this  CRITICAL 

SE  Phase 
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System  Life  Cycle  Model 

(Kossiakoff  &  Sweet) 


Material  Solutions  Analysis 


MIMA 
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Concept  Development 


Operational 

Deficiencies 


Operational 

Requirements 


Derived  •  System  functional 

System  performance  specifications 
requirements  •  Also  SEP/TEMP 


NEEDS  ANALYSIS 

System  Studies 
Technology  Assessment 
Operational  Analysis 


f  CONCEPT 

EXPLORATION 

Concept  Synthesis 
Feasibility  Experiments 
Requirements  Definition 


CONCEPT 

DEFINITION 

Functional  Architecture 
Subsystem  Definition 
Tradeoff  Analysis 


Technological 

Opportunities 


System  Studies 


Candidate  system 
Concepts 


Defined  system 
concept 
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Source:  Kossiakoff  &  Sweet 


Needs  Analysis 

(Kossiakoff  &  Sweet) 

Requirements  Analysis 


Operational 

Deficiencies 


Operational 

Requirements 


Defining  the  System 


Derived  •  System  functional 

System  performance  specifications 
requirements  •  Also  SEMP/TEMP 


NEEDS  ANALYSIS 

System  Studies 
Technology  Assessment 
Operational  Analysis 


CONCEPT 

EXPLORATION 

Concept  Synthesis 
Feasibility  Experiments 
Requirements  Definition 


CONCEPT 

DEFINITION 

Functional  Architecture 
Subsystem  Definition 
Tradeoff  Analysis 


Technological 

Opportunities 


System  Studies 


Candidate  system 
Concepts 


Defined  system 
concept 
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Needs  Analysis 


Functional 

Analysis 


Needs  Analysis 


■  Operations  Analysis  -  Clearly  state  OBJECTIVES 

■  Several  iterations  of  analysis  before  objectives  transform  to 
REQUIREMENTS 

■  Functional  Analysis/  Feasibility  Definition 

■  Objectives  Functions  “Things” 

■  “Physical”  objects  are  initially  logical  abstractions 

■  Assessing  technological  opportunities 

■  Including  production  and  support 


UIMI 
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Architecture  Model  originates  in  Needs 

Analysis 


Needs  Validation 


■  Model-based  operational  effectiveness  analysis 

■  Quantify  the  operational  environment  in  both  normal  and 
“stressing”  conditions 

■  System  performance  parameters  and  constraints  critical  to 
the  model 

■  How  does  the  “new”  system  compare  with  the  legacy  system? 

■  Is  the  need  based  on  overcoming  a  deficiency  or  leveraging 
technology 

■  Outcome  -  Fully  validated  Operational  Architecture  Model 


MSSMk 


Does  the  Functional  Needs  Analysis  result 
from  sound  Concept  SE  practices? 


Concept  Exploration 

(Kossiakoff  &  Sweet) 

Requirements  Analysis  Defining  the  System 


Operational 

Deficiencies 


Operational 

Requirements 


Derived 

System  performance 
requirements 


System  functional 
specifications 
Also  SEMP/TEMP 


NEEDS  ANALYSIS 

System  Studies 
Technology  Assessment 
Operational  Analysis 


f  CONCEPT 

EXPLORATION 

Concept  Synthesis 
Feasibility  Experiments 
Requirements  Definition 


CONCEPT 

DEFINITION 

Functional  Architecture 
Subsystem  Definition 
Tradeoff  Analysis 


Technological 

Opportunities 


System  Studies 


Candidate  system 
Concepts 


Defined  system 
concept 
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Concept  Exploration 


Implementation 

Concept 

Exploration 


Performance 

Requirements 

Validation 


Transform  Operational  to  System  Focus 


■  What  does  the  SYSTEM  have  to  do 

■  Convert  the  Operationally  oriented  view  of  the  system  to 
an  Engineering  oriented  view 

■  Baseline  for  subsequent  phases  of  development 

■  Significant  “exploratory  research  and  development” 

(Kossiakoff  &  Sweet) 

■  This  must  be  completed  BEFORE  system  performance 
requirements  are  quantified 


Discover  and  analyze  critical  issues  and  gain 

insight  into  the  design  task 


(Kroll  et  al) 
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Operations  Requirements  Analysis 


■  Ensure  operational  objectives  are  clear  and  the  requirements 
meet  the  engineering  standards  of  “goodness” 

■  Understanding  compatibility  with  related  Systems  of  Systems 
and/or  Families  of  Systems 

■  Data  and  information  exchanges 

■  CONOPS  is  essential  for  this  phase 

■  If  the  new  system  is  technology  driven,  how  does  the  new 
technology  factor  into  the  CONOPS? 
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Performance  Requirements  Formulation 


■  Achieving  operational  functionality  with  system 
functions 

■  Measurable  Results  of  Value  (RoV) 

■  Conceptual  allocation  of  system  functions  to  abstract 
“Functional  Building  Blocks” 

■  Setting  bounds  of  system  performance  requirements 

■  Design  team  must  set  the  “limits  of  behavior”  (Rechtin) 

■  If  the  RoV  exceeds  the  acceptable  constraints,  a 
“design  trap”  can  result 
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Physical  Implementation  Exploration 


■  “Involves  the  examination  of  different  technological 
approaches,  generally  offering  a  more  diverse  source  of 

alternatives.”  (Kossiakoff  &  Sweet) 

■  Evaluating  concept  alternatives 

■  Setting  parametric  boundaries  and  constraints 

■  Iterating  with  functional  stage 

■  “Bad  or  incorrect  functional  analysis  adversely  affects 
physical  implementation”  (Kroiietai) 

■  Complexity  of  physical  elements  driven  by  functionality 

■  Physical  interfaces  correspond  to  functional  interfaces 
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The  Architecture  Model  begins 
transformation  to  the  Concept  Design 


Performance  Requirements 


■  Performance  Requirements  Validation  process  is  a  “closed 
loop”  process  that  results  in  “system  performance 
characteristics” 

■  Define  WHAT  the  system  must  do 

■  Define  characteristics  in  engineering  terms  that  is  verified 
by  analytical  means  or  experimental  tests 

■  Completely  and  accurately  reflects  the  system  operational 
requirements  and  constraints  including  external  interfaces 
and  interactions 
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Concept  Definition  Stage 

(Kossiakoff  &  Sweet) 

Requirements  Analysis 


Defining  the  System 


Operational 

Deficiencies 


Operational 

Requirements 


Derived  •  System  functional 

System  performance  specifications 
requirements  •  Also  SEMP/TEMP 


NEEDS  ANALYSIS 

System  Studies 
Technology  Assessment 
Operational  Analysis 


CONCEPT 

EXPLORATION 

Concept  Synthesis 
Feasibility  Experiments 
Requirements  Definition 


CONCEPT 
DEFINITION 

Functional  Architecture 
Subsystem  Definition 
Tradeoff  Analysis 


Technological 

Opportunities 


System  Studies 


Candidate  system 
Concepts 


Defined  system 
concept 
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Concept  Definition 


Performance 

Requirements 

Analysis 


Functional 
Analysis  and 
Formulation 
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Conceptual  Design 


■  Concept  transforms  into  a  preferred  solution 

■  Concept  still  involves  sufficient  alternatives,  but  among  the 
choices,  a  final  decision  is  made 

■  The  design  results  from  a  fully  validated  conceptual  design 
model  with  some  preliminary  drawings 

■  Consistent  with  system  performance,  cost  and  schedule  goals 

■  With  acceptable  risk 

■  Fully  considers  support  and  sustainment  -  Total  System 
Effectiveness 
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Cautions  during  Concept  Development 


■  Extreme  Requirements 

■  Meeting  the  requirements  exceed  the  state  of  the 
technology 

■  Meeting  these  extremes  significantly  add  to  cost  and 
schedule 

■  Scope  Creep 

■  Taking  on  too  many  operational  tasks 

■  Adding  scope  during  development 

■  Tightly  coupled  with  Extreme  Requirements 

■  Production 

■  The  production  line  is  usually  just  as  complex  as  the 
system  it  builds 

■  Software  and  test  laboratories 

■  Not  paying  attention  to  Supportability  and  Sustainment 
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Transition  to  TD  and  SDD 


■  Industry  should  be  a  part  of  an  integrated  process 
during  Concept  SE 

■  Each  competitor  will  base  own  concept  model  on  from 
a  single  architecture 

■  Government  SE  IPT  verify  that  developer’s  concept 
traces  to  the  architecture 

■  During  TD  &  SDD,  the  Developer’s  engineering  design 
should  evolve  from  the  concept  design 

■  If  it  doesn’t,  traceability  to  requirements  will  be  difficult 
to  prove 


Transition  to  TD  &  SDD  is  a  major  step,  but  a 
good  architectural  and  conceptual  models  will 

enhance  this  transition 
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Conclusion 


■  “Best  practices”  for  Concept  SE  involves  a  model-based 
design  approach  that  begins  at  Needs  Analysis/ 
Requirements  Definition  and  results  in  the  conceptual 
design  model 

■  Architecture  is  the  basis  for  design 

■  Architecture  is  more  than  just  DoDAF  views 

■  JCIDS  is  a  critical  ENGINEERING  task  where  sponsors, 
requirements  officers  and  project  engineers  work 
together  to  instantiate  the  model 

■  The  artifacts  are  natural  outputs  of  Good  Systems 
Engineering 
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Backup 


Constructing  the  Model 


■  Four  basic  steps 

■  Translating  requirements  into  ideas  into 
understanding 

■  Embedding  the  ideas  into  the  model  to  reflect  the 
requirements 

■  Continuous  iteration  until  the  model  is  sufficient  for 
advancement 

■  Verifying  and  validating  the  model  for  further  action 

■  Continue  the  basic  steps  in  each  stage  of  the  system’s 

life  cycle 


Ref:  Rechtin  and  Jacobs 
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MBSE  -  Initiating  the  Model 


■  MBSE  assumes  the  existence  of  a  well-structured  set  of 

requirements 

■  The  designer  does  not  have  to  know  the  specific  “end” 

■  Only  a  prioritized  understanding  of  variables  that 
can  produce  a  “Result  of  Value” 

■  Modeling  can  assist  the  designer  discover 
requirements  that  are  missed,  misunderstood  or 
overlooked 

■  The  initial  model  is  “rough”  and  often  abstract 

■  But  the  model  facilitates  a  logical  analysis  of  what  will 
become  a  complex  system 

■  Facilitates  stakeholder  discussions  on  future  trade¬ 
offs 
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Source:  Rechtin 


Advantages  of  MBSE 


■  Model  documents  the  evolution  of  the  system  from 
requirements  definition  through  architecture  development  and 
into  conceptual  design 

■  Available  modeling  tools  match  detailed  graphics  with  powerful 
data-bases 

■  Evolution  of  SysML  as  a  standardized  family  of  graphical 
presentations  that  contain  necessary  data  including: 

■  Requirements,  parametrics  and  constraints 

■  Evolution  of  AP-233  standard  for  data  portability  across  models 
and  data  bases 

-  Models  LIVE! 

■  Today’s  “As  Is”  is  the  baseline  for  tomorrow’s  “To-Be” 
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Another  View  of  Concept  Design 


Source:  Kroll  et  al 
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Parameter  Analysis 


■  Parameter  Identification 

■  Examine  all  information  about  the  design  task,  the 
alternative  configurations  that  lead  to  “best  and  final” 

■  Parameters  influence  the  outcome  and  the  optimal 
outcome  may  differ  from  “current  solution  paradigms” 

■  Creative  Synthesis 

■  Craft  a  resulting  concept  that  “solves,  satisfies  and 
embodies  conceptual  parameters.” 

■  Evaluation 

■  Quantifying  strengths  and  identifying  weaknesses 

■  Does  this  system  meet  the  requirements 

■  Is  this  the  right  configuration? 
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Source:  Kroll  et  al 
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SE  for  Concept  Development 


■  Methodical  analysis  from  identification  of  the  initial 
operational  objectives  to  a  validated  concept  design 

■  System  elements  trace  to  operational  elements 

■  Technology  is  feasible  for  advanced  development  and 
engineering  design 

■  JCIDS  Functional  Analyses  is  accomplished  within  the 
scope  of  Concept  SE 

■  SE  Model  originated  in  Needs  Analysis  matures  into 
Concept  model  that  traces  back  to  the  architecture  and 
requirements  models  and  forward  to  the  design  model 
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Aren’t  We  Doing  This  Already? 


■  Yes,  but 

■  Is  Concept  SE  an  integrated  ENGINEERING  activity 
that  includes  requirements  analysis,  architecture 
formation  and  conceptual  design? 

■  Are  the  artifacts  we  develop  during  concept 
development  used  throughout  the  process? 

■  And  are  they  a  suitable  baseline  for  Engineering 
Design 

■  Is  the  Concept  SE  team  employing  MBSE? 

■  If  not,  there  is  likely  a  fundamental 
misunderstanding  of  the  problem  which  correlates 
to  an  incorrect  solution 
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Integrated  T&E  in  a  Joint  Program 

•  Integrated  Test  &  Evaluation  (T&E)  provides  an  integral  part  of  the 
Systems  Engineering  Process,  identifying  levels  of  performance, 
assisting  developers  in  identifying  and  correcting  deficiencies,  and 
validating  to  the  system  owner  that  performance  requirements  are 
met  in  a  cost  efficient  manner.  Historically,  developmental  T&E 
activities  conducted  by  the  Program  Office  have  been  fire-walled 
from  the  operational  T&E  activities  and  organizations. 

•  Joint  Naval  Platform  acquisition  programs  have  the  additional 
constraints  of  supporting  the  needs  and  requirements  of  potentially 
three  varied  customer  groups,  such  as  the  U.S.  Army,  The  U.S. 
Marines,  and  the  U.S.  Navy.  As  the  lead  Program  Office,  NAVSEA 
has  led  the  development  of  processes  and  tools  that  meet  the 
various  programmatic  needs  and  potentially  provide  a  cost  savings 
by  the  use  of  an  Integrated  T&E  environment. 

•  This  presentation  will  discuss  some  of  the  lessons  learned  and  an 
oversight  into  the  methodology  and  tools  used  in  a  program  that  is  a 
model  for  future  joint  programs  to  provide  a  cost-effective  interface 
between  the  Requirements  Engineering,  and  the  Developmental  T&E 
and  Operation  T&E  communities. 


October  22,  2008 


Joint  Acquisition  for  Naval  Platforms 

•  While  many  know  of  the  U.S.  Navy  combatant 
and  non-combatant  fleet  and  of  the  Coast  Guard 
fleet,  most  do  not  know  the  U.S.  Army  maintains 
it’s  own  fleet  of  littoral  non-combatant  vessels. 
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Joint  High  Speed  Vessel  Prototypes 


•  The  U.S.  Army  and  U.S.  Navy  have  been  very 
successful  testing  converted  high  speed  ferries 
as  non-combatant  vessels. 


•  Currently  the  Navy,  Army,  and  Marines  are  jointly 
acquiring  a  production  Joint  High  Speed  Vessel. 

•  NAVSEA  is  the  lead  acquisition  organization. 
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Joint  Requirements 


•  The  three  organizations  formed  an  IPT  to  develop  the 
Analysis  of  Alternatives  (AoA),  Initial  Capabilities 
Document  (ICD),  and  Capability  Development  Document 
(CDD)  IAW  Joint  Capabilities  Integration  and 
development  System  (CJCSI  3710.01). 

•  NAVSEA  coordinated  the  development  and  adjudication 
of  the  AoA,  ICS,  and  CDD,  including  the  Key 
Performance  Parameters  (KPP). 

•  With  it’s  background  of  deepwater  non-combatant  ship 
design,  NAVSEA  took  the  lead  in  the  development  of  the 
platform  Performance  Specification  (PSpec)  and 
coordinated  adjudication  through  the  Joint  IPT. 
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MOE/MOSs,  CTPs  and  COIs 


•  Requirements  module  and  T&E  modules  linked 
by  various  categories  of  measures. 

•  Developmental  T&E  test  events  linked  to  PSpec 
via  Critical  Technical  Parameters  (CTP). 

•  Operational  T&E  test  events  linked  to  CDD  via 
Measures  of  Effectiveness  and  Suitability 
(MOE/MOS). 

•  Additional  concerns  in  regards  to  survivability 
features  and  Live  Fire  Test  &Evaluation  Issues 
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Joint  Test  &  Evaluation  Master  Plan 


•  The  T&E  W-IPT  developed  to  represent  all  the  major  stakeholders. 

•  Stakeholders  include: 

•  Program  Executive  Office,  Ships  (PEO  SHIPS) 

•  NAVSEA  Ship  Design  Manager  (SEA  05D3) 

•  Commander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR) 

•  Army  Test  and  Evaluation  Command  (ATEC) 

•  Marine  Corps  Test  and  Evaluation  Activity  (MCOTEA) 

•  Chief  of  Naval  Operations,  Expeditionary  Warfare  (OPNAV  N85) 

•  Chief  of  Naval  Operations,  Navy  Test  and  Evaluation  Division  (OPNAV  N912) 

•  Deputy  Assistant  Secretary  of  the  Navy  (DASN(Ships)) 

•  Army  Test  &  Evaluation  Executive 

•  U.S.  Army  Test  and  Evaluation  Management  Agency  (TEMA) 

•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics  and 
Technology)  (OASA(ALT)) 

•  Office  of  the  Secretary  of  Defense,  Director,  Operational  Test  and  Evaluation 
(OSD/DOT&E) 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics), 
System  and  Software  Engineering/Assessments  &  Support 
(OUSD(AT&L)/SSES/AS) 
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Integrated  Test  Database 


Supports  Staged  T&E  Approach  through  Acquisition  Lifecycle 


October  22,  2008 
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Web-Enabled  Integrated  Test  Evaluation  Tool 
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SMARTT  ®  Alion’s  Web-enabled  Integrated 
Requirements  Management  and  T&E 


P  ifflegrafed  program  developed  by  Alion  and 
currently  used  in  a  variety  of  internal  and  external 
naval  acquisition  programs. 


October  22,  2008 


Questions  or  comments  can  be  forwarded  to: 


Steve  Randolph,  PMP,  CSEP-Acq 

Program  Manager,  Systems  and  Specialty  Engineering 
Engineering  and  Integrated  Solutions  Sector  (JJMA) 
Alion  Science  and  Technology 
4300  King  Street 
Alexandria,  VA  22302 
srandolph@alionscience.com 

steverandolph@cox.net 
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Capitalizing  in  migrating  web  service 


environments  requires  focused 
diligence  in  tactical  and  strategic 
considerations  in  achieving  Net- 
Centric  efficiencies  and  operational 
utility. 

Net-Centric  strategies  present 
challenges  and  not  easily  integrated 
into  engineering,  acquisition, 
testing,  management,  and  funding 
disciplines. 

</SCRA 
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Applying 


'Adopt  or  Buy'  (adapt  and  use)  strategies  to  existing 
web-services  to  meet  acquisition  criteria  may 
ignore  or  delay  essential  'business  rules'  and  use; 
thereby  not  exploiting  technologies  for  greater  net- 
centric  capability  end-goals  in  the  field. 


Evaluating 


a  single  or  group  of  web-services  in  a  transitioning 
environment  may  well  stovepipe  web-services  as 
system/system  function  replacement  and  focus 
testing  on  program;  which  yield  less  than  optimum 
net-centric  operational  efficiencies. 


Deploying 


web-services  without  exploitation  of  the  web- 
service  in  a  given  mission-to-task  consideration  may 
hinder  product  operational  usage  and  foster  miss- 
use  or  non-use. 
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Challenges 


Focus  on  urgent 
operational  need  - 
solution 

stakeholders  must 
forge  a  single 
'integrated' 
enterprise  to  reduce 
risk  in  satisfaction  of 
that  need. 


Changing  the  Business  Model  Requires: 

( 1 )  Willingness  to  empower  teams  working  together  to  achieve  more  than 
organizations  working  alone 

(2)  Focus  on  Operator  or  Warfighter  as  central  driver  -  solution  need  originator  and 
evaluator 

(3)  Commitment  to  providing  meaningful  services  rather  than  inflexible  " products " 


- 7 
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Familiar 

Less  familiar 

What  we  use 

FOCUS 

What  we  use  and  how  we  use  it 

Technology  affects  on  system 

SOLUTION 

Technology  +  method  +  people 

capability 

affect  on  operational  capability 

Developers'  perspective 

PERSPECTIVES 

Warfighter  perspective 

Hardware  and  software  must  be 

CENTRAL  RULE  or 

Materiel  and  non-materiel  must  be 

developed  together 

CONCEPT 

developed  together 

SoS  assessment 

APPROACH 

MCP  assessment 

-  OT&E  focus  on  the  system 

-  Holistic  focus  on  all  components 

System  centric 

CENTRICITY 

Capability  centric  (Warrior) 
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(gllobarottng  To  Advwica  Technology 


©2008  By  SCRA.  All  Rights  Reserved 


3 


Changing  Business  Model 


CAPITALIZING  REQUIRES 
KNOWING  THE  CENTER 
OF  GRAVITY  (COG)  OF  THE 
PROBLEM  THAT  YOU  ARE 
TRYING  TO  SOLVE 


^SCRA 

GdUmfel  To  Advance  Technology 
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Point  1 


CAPITALIZING  REQUIRES 
MEETING  MUTUAL 
INCLUSIVE  PERSPECTIVES 
-  PROGRAM  MANAGERS, 
DEVELOPER,  TESTER,  AND 

END  USER 


V'SCRA 
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CAPITALIZING  AND 
PROVIDING  A  NET-CENTRIC 
MISSION/TASK  CAPABILITY 
REQUIRES  INTEGRATING 
THREE  INSTRUMENTS: 
TECHNOLOGY,  PROCESS, 

AND  PEOPLE 

SCRA 
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Point  3 


"Coevolved  MCP  (Mission  Capability 
Packages)  response  to  the  problems 
that  can  arise  when  new 
technologies ...  are  introduced  but  are 

not  accompanied  bv  changes  jn  other 
areas,  such  as  training  or  doctrine" 
For  years,  the  belief  has  been  that 
"computer  technology  was  not  cost- 
effectives  because  there  was  a  lack  of 
empirical  evidence  to  show 
improvement  in  productivity." 


t/SCRA 
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Net-Centric  Operations 


Capability  Maturity 


Meeting  the  challenges  require  integrating  view  point  ana  instruments  tnrougn 
progression  of  experimentation,  integrated  testing,  and  exercises  &  operations. 

V  ".\  l  ^  - 
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Requirements 
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Technology  Maturity 
Requirements 
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Acquisition 
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Capability 


•  Innovative  Brokering 

•  Experimentation 


•  Assessments 

•  Integrated  Testing 
•DT/OT 

•NR-KPP  Certification 


Exploring  concepts 
and  technology 
readiness  ^ 


Meeting  criteria  and 
operations 
interoperability  (use) 


Customer/User 


Maturity  of  Useful 
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Exercises  and  Operations 


Addressing  command  issues  and 
demonstrating  joint  capability 
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Integrated  Capability  Maturity  Model 


An  Operational  Mission  Thread  is  an 
operational  and  technical  description 
of  the  end  to  end  set  of  activities  and 
systems  that  accomplish  the 
execution  of  a  mission 


Download 
BDA  Images 


Conduct  BDA  jjp 
Decide  on  Restrike 


JFLCC< 


k  Approve 

< - 

*  Plan 

- > 

>  JFACC 


Exploit 
BDA  Images 


DCGS 


Both  operational  and  technical 
products  are  used  to  document  a 
mission  thread 


Targeting  Mission  Thread 
includes  part  of  the 
Collection  Management 
Mission  Thread  to  provide 
the  set  of  end  to  end 
activities 


VSCRA 
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Operational  Mission  Thread  (OMT)  Definition 


Modeling 
and  Table 
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Engagement  Opportunities  -  How  do  you  Know? 
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Engagement  Opportunities  -  What  does  this  mean? 


ran  you  sho*  *at  th“ 
«mand  priori^ issueh* 

t0mCnS““^“'W 

addressed? 
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Engagement  Opportunities  -  Can  you  show  me? 


Exercise  <  'Testing  &  Assessment 


Critical  Questions 

Measures/ 

Metrics 

Focus 

Traditional  Approach 

Do  services  provide  html/xhtml  display  of 
Blue  Force  friendly  forces  location? 

Information 

Display 

Individual  solution/services 
(technical  specification) 

Developmental  Testing 

Does  information  exchange  between 
services  (solutions)  comply  with  message 
format  and  standards  (i.e.,  XML,  NR-KPP, 
security)? 

Response  time. 
Transition  load, 
and  Web-service 
availability 

Between  solution/services : 
Technical  interface  level 
(technical  specification) 

Developmental  and 

Technical  Interoperability 

Do  services  (solution)  meet  information 
exchange  specifications? 

Message  Format 
Standards  and 
data/Taxonomy 
standards 

Individual  solution/services 
(technical  specification) 

Developmental  and 

Technical  Interoperability 

Do  services  provide  access  to  and  display 
friendly  force  location  from  automated 
track  feeds? 

Requirement 

Statement 

Individual  solution/services 
meet  requirements 

Operational  Testing 

Do  services  provide  access,  generate,  and 
display  overlay  information? 

Requirement 

Statement 

Individual  solution/services 
meet  requirements 

Operational  Testing 

Do  group  of  services  improve  Common 
Picture  Overlay  interoperability? 

Priority  C2  Related 
Issue 

Group  of  material  solutions, 

C2  environment,  and 
business  rules 

Operational  Testing, 
Assessments 

Does  group  of  services  increase  Common 
Picture  Track  Management  Capability? 

Priority  C2  Related 
Issue 

Group  of  material  solutions, 

C2  environment,  and 
business  rules 

Operational  Testing, 
Assessments 

Do  solutions  address  C2  System 
Interoperability  for  DoD,  Coalition,  Multi¬ 
national,  Agencies,  and  NGOs 

Command  or 

Theater 

Operations  Issue 

Group  of  MCP  solutions,  C2 
environment,  and  business 
rules 

Assessments,  Exercises 

Do  solutions  increase  Joint  net-centric 
operations  with  interagency, 
multinational,  and  operational  forces 

Command  or 

Theater 

Operations  Issue 

Group  of  MCP  solutions,  C2 
environment,  and  business 
rules 

Assessments,  Exercises 

1/SCRA 
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Level  of  Enabling  Questions:  Testing  and  Exercise 


•  Capitalizing  requires  knowing  the  Center  of 
Gravity  (CoG)  of  the  problem  you  are  trying  to 
solve 

•  Capitalizing  requires  meeting  mutual  inclusive 
perspectives  -  program  manager. ;  developer, 
tester,  and  end-user 

•  Capitalizing  and  providing  a  net-centric 
mission/task  capability  requires  integrating  three 
instruments:  Technology,  Process,  and  People 

•  Drive  capitalization  with  appropriately  timed 
4 engagement *  questions 


^SCRA 
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SUMMARY 


Joint  Surface  Warfare 
Joint  Capability  Technology 
Demonstration  - 

Maturing  Weapon  Data  Link  Concepts  into 

Operational  Capability 


Robert  Finlayson 
Senior  Systems  Engineer 


NAVAIR  Public  Release  SPR-08-924 
Distribution  Statement  A  -  Approved  for  public  release;  distribution  is  unlimited 
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Demonstration  Description 


•  Developing  a  capability,  not  a  system 

-  System  of  systems  approach 

•  Leverages  maturing  weapon  data  link  network  technologies 

-  Demonstrate  the  integration  of  multiple  Intelligence,  Surveillance,  and 
Reconnaissance  (ISR)  and  launch  platforms  with  existing  stand-off  weapons 

-  Allows  interchangeable  ISR  assets  to  provide  initial  targeting  data  and  in¬ 
flight  target  updates  for  multiple  weapons 

-  Provides  multiple,  comprehensive  joint  kill  chain  threads  to  the  Combatant 
Commander 

-  Significantly  increases  operational  agility 

-  Increases  probability  of  target  kill  in  adverse  weather  conditions  and  at 
extended  ranges 

-  Minimizes  launch  platform  threat  exposure 


•  Conscious  decision  to  organize,  plan  and  execute  demonstration  as  if  it 
were  a  program 

-  Programmatic  and  system  engineering  discipline 


dPL 
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JSuW  Background 


•  In  FY07  Advanced  Concept  Technology  Demonstrations  (ACTD)  were  re¬ 
designated  to  JCTD 

•  Managed  out  of  PMA  201  -  Precision  Strike  Weapons 

•  JSuW  approved  for  FY07  start 

-  Kickoff  in  June  2007 

•  Approximately  three  year  period  of  performance  and  $40M  effort 

•  Follow-on  to  the  Weapons  Data  link  Network  ACTD 


•  JSuW  involves  five  programs  of  record  (PoR) 

-  Joint  Standoff  Weapon  (JSOW-C-1),  Harpoon  Block  III  and  F/A-18E/F  are  funded  for 
J1 1  message  integration  as  part  of  their  PoR 

-  Joint  Surveillance  and  Target  Attack  Radar  System  (JSTARS)  and  P-3C  Littoral 
Surveillance  Radar  System  (LSRS)  will  incorporate  J 1 1  for  demo  purposes  only 


dPL 


3 


NAVAIR  Public  Release  SPR-08-924 

Distribution  Statement  A  -  Approved  for  public  release;  distribution  is  unlimited 


Technical  Implementation 


•  Incorporate  the  J 1 1  message  software  into  existing  Link-16  terminals 

•  Interim  Change  Proposal  to  Link-16  (MIL-STD-6016C) 

-  J1 1.1  Directive  messages 

•  Sent  to  the  weapon 

-  J1 1 .0  Status  Response  messages 

•  Sent  from  the  weapon 

-  J1 1 .2  Weapon  Coordination  messages 

•  Coordination  of  NEW  control 

•  Sent  and  received  by  weapon  controllers  and  In-Flight  Target  Update  (IFTU) 
Third  Party  Sources  (3PS’s) 

•  Weapons  are  receiving  the  Strike  Common  Weapon  Data  Link  radio 

-  Rockwell  Collins 
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Operational  View 


Integrate  the  Link-16  J11 
Message  Set  into  existing 
software  architectures  for 
the  JSTARS  and  LSRS 
platforms 

Ensure  interoperability 
with  the  JSOW-C-1, 
Harpoon  Block  III,  and 
F/A-18E/F  programs  of 
record  (incorporating  J11 
message  set) 

Develop  the  associated 
CONOPS/TTPs 
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Concept  of  Operations 


•  F/A-18E/F,  third  party  targeting  source  (3PS;  i.e.,  second  F/A-18E/F,  JSTARS, 
LSRS)  or  other  ISR  platform  detect  enemy  combatants 

•  J1 1 .2  messages  passed  between  controller  /  shooter  (F/A-18E/F)  and  3PS  for 
coordination 

•  Weapon  released  by  shooter  (F/A-18E/F) 

•  3PS  provides  In-Flight  Target  Updates  (IFTUs)  to  weapon  via  J1 1 .1  messages 

•  Weapon  replies  with  Weapon  In-Flight  Track  (WIFT),  Ack/Nack  and  Bomb  Hit 
Indication  via  J1 1 .0  message 
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PM-SE  Interaction 
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Organizational  Breakdown  Structure 


JSuW  JCTD 
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Schedule 


Executive 
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Operational 
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Test  and 
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Assessment 


JCTD  Deputy 
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_ Integration _ . 


Requirements  & 
Architectures 


Warfare  MS&A 
and  Risk 


Training 


Logistics 


Cost 


Scenarios 


CONOPS 


Mission  Planning 


Test  Planning 


Test  Execution 


Data  Collection 


Metrics 


"  Tactics, 
Techniques 
&  Procedures 


Assessment  & 
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NEWIWG 


JC2NEW 

JT&E 


MIL  STD  6016C 
ICP 


Platform 

Configuration 

Management 
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Setting  Constraints 


Organizational  Breakdown 


Integrated  Master  Plan 


Work  Breakdown  Structure 


Integrated  Master  Schedule 
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Capability  Statement 


Challenge:  Cost  effective,  simultaneous,  multi-axis  strike  in  the 
littorals,  against  a  mutually  supported,  state-of-the-art  surface  action 
group  (SAG);  at  the  time  and  place  of  our  choosing 


dPL 
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Defining  Scenarios 


Understand  the  environmental 
conditions 

-  Use  a  guide  to  ensure  all  potential 
impacts  have  been  addressed 

Look  at  a  range  of  scenarios 

-  Address  each  mission 

-  Across  the  spectrum  of  “easy  to  hard” 

Understand  the  requirements  and/or 
desired  capabilities  for  each 
scenario 

-  How  does  this  affect  system  design 
and  performance? 

Distribute  demonstration  resources 
to  address  the  scenario  spectrum 


LI  Land 
1.2  Sea 
L3  Air 
L4  Space 


2.1  Miamian 

2.2  Forces 

2.3  Command,  Control, 

St  Communications 

2.4  Intelligence 

23  Deployment,  Movement 
&  Maneuver 

2.6  Firepower 

2.7  Protection 
2,3  Sustainment 
23  Threat 
2.10  Conflict 


Figure  C-l.  Organization  of  Conditions  for  Joint  Tasks 


3.1  Political  Policies 

3.2  Culture 

3.3  Economy 


CJCSM  3500.04D,  Universal  Joint 
Task  List,  1  August  2005 


-  Engineering  level  analysis,  modeling 
and  simulation,  flight  test,  etc. 


li 
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Capability  Decomposition 


Capability 

Attribute  / 
Requirement 

Sensor 

Shooter 

Controller 

Weapon 

External 

Sources 

Network  / 
Message 

Cost  Effective 

Efficient  use  of 
assets 

Minimize  standoff 
sensors 

Increase 

survivability 

Co-locate  with 
shooter/sensor 

Level  of  effort 
weapons 

Low  cost  intel. 

collection 

platforms 

Use  existing 
network 

Simultaneous 

Coordinated 
(timing,  position) 

Synchronized  with 
shooter  and 
weapon 

Synchronized, 

positioned 

Auto-logic; 
advanced  USI 

Predictable/ 
programmable 
flight  profile 

Multiple, 

dispersed 

collection 

Number  of 
users 

Multi-Axis 

Pre-planned 

Position  wrt 
shooter/weapon 

360  LAR  wrt 
target 

Controller 

positioning 

Maneuver  in 
flight 

Multiple, 

dispersed 

collection 

Range 

Strike 

Kinetic  attack 

Targeting  wrt 
weapon 

Loadout, 
weapon  support 

ROE  feed; 
combat  ID 
assurance 

Lethality  vs 
target  set 

Multi-role 
platforms; 
collect  and 
strike 

Detailed 
message  set 

Littorals 

Clutter,  neutral 
shipping 

Resolution, 
accuracy,  fusion 

Range  from 
base,  CVN 

Advanced  SA 

Selectivity,  Al, 
scan  volume 

Deployable; 

survivable 

Spectrum 

management 

SAG  -  Mutual 
Support 

Integrated  air 
defense  system 

Standoff,  fusion 

Standoff 

Standoff 

Survivability 

Survivable 

Range 

SAG  -  SOTA 

Stealth,  CCD, 
decoys,  firepower 

Accuracy,  fusion, 
jam  resistant 

Situational 

awareness 

Advanced 
decision  tools  - 
superior  SA 

Selectivity, 

CCM,  Al,  Jam 
resistant 

Embedded 

artificial 

intelligence 

Resilient 

Time  and  Place 
of  our  Choosing 

Independent  of 
environment 

All  Weather  (vis, 
sea  state,  etc.) 

Endurance 

Endurance; 
comm  links 

Detect  target 
in  all  weather 

Persistent 

Reliable 
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System  Performance  Measures 


Metrics 

^-^ntities 

Extent 

Accuracy 

Timeliness 

Reliability 

Robustness 

Sensor 

#  targets 

Range 

TLE 

Update  rate 
Resolution 

Internal  latency 

IFTU  rate 

MTBF 

ETOS 

Turn  time 

Survivability 

Discrimination 

Jam  effects 

Shooter 

#  weapons 

Sensor  range 

Launch  envelope 

Msg  processing 

HSI 

Platform  speed 

MTBF 

ETOS 

Sys.  Architecture 

Survivability 

Launch  envelope 

Weapon 

Range 

Flight  profile 

Seeker  res. 

Control  logic 

Aero  perf. 

IFTU  processing 

Speed 

Loiter  ability 

MTBF 

WIFT  trans. 

Env.  Effects 

Survivability 

Discrimination 

Network 

Range 
#  JUs 

Bandwidth 

Msg.  transfer 

Mission  planning 

Latency 

Aircraft  interface 

Packet  loss 

MTBF 

Protocols 

Jam  effects 

Encryption 

Controller 

#  weapons 

#  targets 

IFTU  rate 

Data  fusion 

HSI 

Internal  latency 

MTBF 

Location 

Tgt.  Processing 

Jam  effects 

External 

Sources 

#  available 

Gateway 

Network-network 

latency 

Data  security 

Intel  fusion 

Network  access 

Msg.  format 

dPL 
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Interoperability  Challenge 


Link-16  (MIL-STD-6016C,  Interim  Change 

Proposal  TM06-093Ch2) 

-  Approved  by  the  Joint  Multi-TADIL 
Configuration  Control  Board  (JMTCCB)  on  02 
May  2008 

-  Staffing  underway  for  NATO  review 

-  Message  standard  is  still  in  “interim  state” 

Using  Excel  spreadsheets  for  interoperability 

assessment  and  configuration  management 

-  Awaiting  Interoperable  Systems  Management 
and  Requirements  Transformation  (iSMART) 
configuration  change  to  the  ICP 

-  Compare  each  platform’s  implementation  by 
software  version 

-  Identify  interoperability  gaps  and  work  with 
platform’s  to  eliminate  discontinuities 

Migrate  eventually  to  iSMART  as  well  as 

MS&A  tools  currently  under  development 


J11.0  Word  Definitions 

(Your  Name  Here)  3PS  Implementation 

Transmit  Implementation 

Maturit 

Receive  Implementation 

Maturity 

Message  Use 

1  2 

3 

4  5 

6 

7 

8 

1  9 

1  2 

3 

4  5 

6 

7 

8  1 

9 

Level 

Weapon  Status  Discrete  Value 

1  2 

7 

9  10 

3 

8 

5 

6  4 

1  2 

7 

9  10 

3 

8 

5 

6  4 

2  1 

O  o 

■§  E 
o  o 

Change  2  Version 

ACK 

NACK 

BHI 

Basic  WIFT 

WIFT  Supplement 

Ping  Response 

Self  Abort 

Contact  Report 

Handoff  Checkin  -  Dire 

Handoff  Checkin  LostC 

* 

<  z 

BHI 

Basic  WIFT 

WIFT  Supplement 

Ping  Response 

Self  Abort 

Contact  Report 

1  1 
i  i 

J11.0I 

WEAPON  RESPONSE/STATUS  INITIAL  WORD 

N/A 

i 

270 

271 

800 

704 

700 

NEW98 

1664 

1664 

387 

4085 

358 

1107 

1107 

1107 

1107 

1107 

1107 

001  WORD  FORMAT 

004  LABEL,  J-SERIES 

005  SUBLABEL,  J-SERIES 

001  MESSAGE  LENGTH 

NEW20  WEAPON  STATUS  DISCRETE 

NEW18  TYPE  OF  NEW 

NEW99  WEAPON  PROFILE 

NEW44  FUZE/PAYLOAD  SUBSYSTEM  STATUS 

NEW39  MISSION  PROCESSOR  STATUS 

NEW40  IMU  SUBSYSTEM  STATUS 

NEW41  GPS  SUBSYSTEM  STATUS 

NEW42  TERMINAL  GUIDANCE  SUBSYSTEM  STATUS 

NEW43  PROPULSION/CONTROL  ACTUATOR  SUBSYSTEM  STATU: 
NEW13  BASE  TIME 

NEW38  NAV  MODE 

NEW66  PREPLANNED/ACTIVE  MISSION  INDEX  NUMBER 

NEW152  SEEKER  ACQUISITION  CONFIDENCE 

NEW37  JDPI/MISSION  NUMBER  INDICATOR 

NEW30  CONTROLLER  COMMUNICATIONS  INDICATOR 

NEW31  ALTERNATE  CONTROLLER  COMMUNICATIONS  INDICATO 
NEW32  THIRD  PARTY  COMMUNICATIONS  INDICATOR 

NEW33  THIRD  PARTY  IFTU  ENABLED  INDICATOR 

NEW34  LAUNCH  PLATFORM  CONTROL  INHIBIT  INDICATOR 

S 

R 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

0  0 

0  0 

1  2 

0 

11 

0 

7 

9  10 

0 

0 

3 

0 

11 

0 

8 

0 

0 

5 

0  0 

0  0 

6  4 

4077 

NEW295  TERMINAL  GUIDANCE  TYPE 

756 

004  |  SPARE 
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JSuW  T&E  Strategy 


Network 

Constructive 

Sim 

Overall  Scenario 


Msg  Virtual 
Constructive 
Sim 

•  Compliance  to  ICP 


Design  Feedback 


SIMEX 
07  &  09 

•CONOPS  and 
TTPs 

•SoS 

connectivity 


Platform  Msg 
Development 
And  Testing 


Lab 

Testing 

•Interoperability 
•Isolated  / 
Distributed  env. 
•  Pre- 
production 
Software 
•Rep 
Simulate 


Military 

Utility 


ment 


j 


Ground 

Testing 

•  Production 
software 

•  Actual 
Hardware 
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Tactics,  Techniques  and  Procedures 
Development 


Maturing  TTPs  through: 

-  Engagement  Simulation 

-  Table  top  role  play 

-  Simulation  Exercise 


•  Constructive,  virtual 

-  Ground  demo 

-  Flight  demo 


•  Balance  demo  ops  with  real  world  CONOPS 
development 

-  Scenario  dependent,  design  to  succeed 

•  Continual  trade-off  and  maturation  of  TTPS 
in  parallel  with  message  set  implementation 

•  Validation  and  modification  with  demo  (T&E) 
activities 


dPL 
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Risk  Management  Board 


Chairman 


SE  WG  Lead 


Technical  Director 


Test  Director 


JCTD  Deputy 


Platforms 


RISK  ID 
Lead  Subgoup 
Last  Updated 

RISK  DESCRIPTION 

Risk  Statement: 


RISK  TITLE: 


Name 

Risk  Manager  Info  Phone 


Root  Cause: 

Identified  By: 

RISK  ANALYSIS 

Risk  Type 


Risk  Indicators/Notes: 


%  Occurrence  Probability 
Probability  Chart  Value 


CURRENT  RISK 
ASSESSMENT 
SUMMARY 


m 


RISK  COMPARISON 

Ranking: 

0 

1  2  3  4  5 

Impact 


RISK  MITIGATION 


Action  /  Event 

Date 

Success  Criteria 

%  Occurrence 
Probability  and 
Impact  Value 
(if  successful) 

POC 

Status 

Scheduled 

Actual 

1 

2 

3 

Working  Group  Reps 


JSTARS 


Harpoon 


Performance 

Assessment 


Operational  Utility 


System  Engineering 
and  Integration 


Transition 


dPL 
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Interoperability  Certification  Proposal 


•  Can  JSOW-C-1  and  Harpoon  Block  III  use  the  JSuW  demonstration 
events  to  obtain  certification? 

-  Save  $$ 

-  Improve  understanding  of  NEW  certification  process 

-  Streamline  test  planning  and  execution 

-  Develop  a  process  for  certifying  future  Net  Enabled  Weapons 
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JITC  Certification 


19 


Interoperability  Test 
Certification  Process 


^-"■N  on  -Cere?  ICTOT^ 
Jnterim?  Req’s  Chgd?^ 
Expired? 


Valid 

Certification 


Cert 

Required 
•sby  J-6?^ 


Req's 

hanged^ 


Change 

ICEP? 


JITC  Certification 


Identify 

Requirements 


Develop  Cert. 
Approach/ICEP 


Perform 

Evaluation 


Full  Limited 
Certification 


Special 

Certification 


Req's 

Certified 

J-6?^ 


Standards 
Conform  a  nee 


NR-KPP, 
Arc  hitectu  res, 
I  A,  ...  Req’s 


Stds  Conform, 
KIPs,  NR-KPP 

.  Assessment 


Test 

Requirements/ 

Capabilities 


No  Certification 


JJTC 


JITC 


JITC 


(Optional) 

JITC 

Memorandum 

“'Ho  Certification 
Requi  red” 


Figure  E-l.  Interoperability  Test  Certification  Process 


CJCSM  6212.01D,  Interoperability  and  Supportability  of  IT  and  NSS,  14  March  2007 
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Summary 


•  Joint  Surface  Warfare  JCTD  has  provided  a  challenging  systems 
engineering  environment 

-  Engineering  a  capability  more  than  a  system 

-  Team  dispersion 

-  Requirements  allocation 

-  Interoperability  assurance 


•  Programmatic  and  SE  discipline,  practices  and  procedures  still  apply 

-  Demonstrations  don’t  give  you  a  “free  pass”  when  it  comes  to  project 
management  and  engineering 


•  Expect  more  of  the  same  in  the  coming  decades 

-  Unmanned  system  expansion 

-  Weapon  maturity  and  migration 

-  Adaptation  of  CONOPS  and  TTPs  to  optimize  NEW  capability 
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GENERAL  DYNAMIC 

Land  Systems 


EXTENDING  ENTERPRISE  SYSTEMS  FOR 
AN  INTEGRATED  LOGITICS 
MANAGEMENT  ENVIRONMENT 

1 1th  Annual  Systems  Engineering 

Conference 


Presenter:  Mike  Korzenowski 
Oct  20th,  2008 


Authors:  GLSN  Team  -  Kurt  Hansen,  Jim  Garrity 


Introduction 


•  A  System  Engineering  approach  wrapped 
with  a  Design  For  Six  Sigma  (DFSS)  blanket 
of  methodology  to  provide  a  means  of 
designing  and  delivering  an  Integrated 
Logistics  Management  Environment  for  the 
collaboration  and  delivery  of  logistics 
information  over  a  military  support  network. 


GENERAL  DYNAMIC5 

Land  Systems 
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Large  Scale,  Sprawling  Systems 


•  Stove  pipes  for  product  source  and  delivery 

•  Security,  Limited  access 

•  Funding  Problems 

•  Heavy  Payloads,  Quick  Access  Demands 

•  Heavy,  Traditional  Process  Driven  methods 

•  Old,  New  Mil  Standards 

•  Large  Complex  Legacy  Systems 

•  New  Technologies  waiting  to  be  exploited 


GENERAL  DYNAMIC5 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-98,  dated  10/13/08  3 


Transitions  to  Deal  With 


•  Old  Technology,  Legacy  Processes 

•  Logistics  maintainability  models 

•  Today's  technology  without  changing  the 
development  processes 

•  Streamlined  delivery  over  Global  Support 
Networks 

•  Containment  to  Military  Networks  with  limited 
access 


•  Low  time,  cost 


•  High  Demand,  The  Right  Data  at  the  Right  Time 


GENERAL  DYNAMIC5 

Land  Systems 
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The  Typical  Full  Product  Life  Cycle 

|  Concept 

Feasibility  Project  Definition  Development  Production 

In  Service 

Disposal 

Logistics  Product  Data 
Management  (LPDM) 

/Systems 
Logistics 
Operations 


GENERAL  DYNAMIC 

Land  Systems 
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(ILS)  Has  not  changed 
1388-2B  MIL-Standard  (circ  1973) 


GENERAL  DYNAMIC! 

Land  Systems 
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ILS  -  Integrated  Logistics  Support 


•  The  ILS  management  process  which  facilitates  development  and 
integration  of  the  10  individual  logistic  support  elements  to  specify, 
design,  develop,  acquire,  test,  field,  and  support  systems.  There 
are  10  ILS  elements: 

7i  Maintenance  planning 

7i  Supply  support 

7i  Support  and  Test  Equipment/Equipment  support 

7i  Manpower  and  personnel 

7i  Training  and  training  support 

7i  Technical  data 

7i  Computer  Resources  support 

7i  Facilities 

7i  Packaging,  Handling,  Storage,  and  Transportation  (PHS&T) 

7i  Design  interface 


GENERAL  DYNAMIC5 

Land  Systems 
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Source  Data  Model  and  Outputs  For 
Supportability  (ILS  Program) 


SYSTEM  PERFORMANCE 


GENERAL  DYNAMIC5 

Land  Systems 
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Products  Delivered 


•  Technical  manuals. 

•  Technical  and  supply  bulletins. 

•  Transportability  guidance  technical  manuals. 

Maintenance  expenditure  limits  and  calibration  procedures. 

•  Repair  parts  and  tools  lists. 

•  Maintenance  allocation  charts. 

•  Preventive  maintenance  instructions. 

•  Drawings/specifications/technical  data  packages. 

•  Software  documentation. 

•  Provisioning  documentation. 

•  Depot  maintenance  work  requirements. 

•  Identification  lists. 

•  Component  lists. 

•  Product  support  data. 

•  Safety  critical  parts  list. 

•  Lifting  and  tie  down  pamphlet/references. 

•  Hazardous  Material  documentation. 


GENERAL  DYNAMIC5 

Land  Systems 
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Typical  Tools  -  Program  to  support  ILS 


DEPLOYMENT 


•Adobe  PDF 
•EMS2  Runtime 
•InstallShield 
•Internet  Explorer 
•Powerarchiver/ 
Winzip 
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The  Enterprise,  High-Level  Military 


GENERAL  DYNAMIC5 

Land  Systems 
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The  Enterprise,  High-Level  Military 


GENERAL  DYNAMIC 

Land  Systems 
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Why?  Products  Delivered  over  Network 


•  Data  is  now  capable  of  streaming  and  is 
considered  developed  in  native  format  for 
delivery  over  a  network,  (serialization) 

•  Near-time  access  is  required,  updates  as  well. 

•  Publish-Subscribe  methods  is  desired,  only 
when  I  need  it  mentality 

•  Authentification,  Security  maintained  easily 

•  Information  Assurance  can  be  applied 


•  Feedback  to  OEM 


GENERAL  DYNAMIC5 

Land  Systems 
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Attempts  to  Solve 


•  Replacing  legacy  systems  with  integrated  COTS  packages  (like  Baan,  PeopleSoft,  and  SAP) 

•  Developing  data  and  information  warehouses 

•  Establishing  central  operational  data  stores  or  data  clearinghouses 

•  Implementing  Enterprise  Portals 

•  Using  Middleware 

•  Using  XML 

•  Reengineering  all  applications  to  a  single  architecture 

•  All  of  these  approaches  have  value  and  some  will  even  provide  at  least  temporary  benefit. 
However,  unless  they  are  business-driven  and  model-based  they  are  more  likely  to  further 
compound  the  problems  than  provide  a  solution. 


GENERAL  DYNAMIC5 

Land  Systems 
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Design  for  Six  Sigma  (DFSS) 


•  The  goal  of  DFSS  is  to  create  designs  that  are 
development  efficient,  capable  of  exceptionally 
high  yields  and  are  robust  to  process 
variations. 


GENERAL  DYNAMIC5 

Land  Systems 
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Design  for  Six  Sigma  (DFSS) 


•  Capture  Voice  of  Customer  &  Define  Eng.  Requirements 


71 

Wants  &  needs  tools 

71 

Customer  use  observations 

71 

Kano  Analysis 

71 

Quality  Function  Deployment 

(QFD) 

Develop  Concepts  and  Select 

71 

Pugh  Matrix 

71 

Axiomatic  Design 

71 

TRIZ 

71 

Failure  Mode  &  Effects  Analysis  (FMEA) 

•  Develop  Detailed  Design 

71  Systems  Engineering 

71  Function  Models  &  FMEAs 
71  T ransfer  Functions 

•  Statistical  Design 

71  Monte  Carlo  Analysis 

•  Design  for  Robust  Performance 

71  Design  of  Experiments 

71  Robust  Design 

71  Design  for  Reliability 

•  Design  for  Manufacturability 

71  Process  Capability  Databases 
71  Statistical  Tolerancing 

•  Predict  Quality 

7i  DFSS  Scorecards 


GENERAL  DYNAMIC5 

Land  Systems 
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Axiomatic  Design 


•  Axiomatic  design  is  systems  design  methodology  using 
matrix  methods  to  systematically  analyze  the 
transformation  of  customer  needs  into  functional 
requirements,  design  parameters,  and  process  variables. 


GENERAL  DYNAMIC5 

Land  Systems 
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2  Principles  of  Axiomatic  Design 


•  Axiom  1:The  Independence  Axiom 

71  Maintain  the  independence  of  the  functional 
requirements  (FRs) 

•  Axiom  2:The  Information  Axiom 

7i  Minimize  the  information  content  of  the  design 


GENERAL  DYNAMIC 

Land  Systems 
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Key  Axiomatic  Design  Definitions 


•  Customer  Needs  (CN) 

71  Collection  of  statements  expressed  in  the  “voice  of  the  customer”  that  express  the  customers’ 
perceptions  of  the  design  task 

•  Functional  Requirement  (FR) 

71  Minimum  set  of  independent  requirements  that  completely  characterize  the  functional  needs  of 
the  product  (or  software,  organizations,  systems,  etc.)  in  the  functional  domain 

•  Constraint  (C) 

71  Bounds  on  acceptable  solutions 
71  There  are  two  kinds  of  constraints: 

■  Input  constraints 

71  Imposed  as  part  of  the  design  specifications 

■  System  constraints 

71  Imposed  by  the  system  in  which  the  design  solution  must  function 

•  Design  parameter  (DP) 

71  Key  physical  (or  other  equivalent  terms  in  the  case  of  software  design,  etc.)  variables  in  the 
physical  domain  that  characterize  the  design  that  satisfies  the  specified  FRs. 
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PROCESS  VARIABLES 


Three  States  of  Functional  Coupling 


Uncoupled 


Decoupled 


Coupled 


Each  DP  uniquely 
satisfies  a  single  FR 

Order  of  Development 
and  Function  not 
important 


Some  DP’s  impact  more 
than  one  FR. 

A  Progressive  Solution  is 
possible 

Order  of  Development  and 
Function  are  important 


Some  DP’s  impact  more 
than  one  FR. 

A  Simultaneous  Solution 
is  required 

Order  of  Development  and 
Function  are  important 
and  will  require  iterations 
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Axiomatic  design: 

Evaluate  options  using  the  independence  axiom 


The  Independence  Axiom: 

Maintain  independence  between  functional  requirements 


Decoupled  -  Acceptable 


FR.1 

X 

0 

0  -A 

FR.2 

X 

0 

FR.3 

X 

* 

X 

Uncoupled  -  Desired 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-98,  dated  10/13/08 


Analysis 


•  Analysis  from  DFSS  axiomatic  design  methods  indicate  the  need  for  a  point-point 
solution,  (eliminate  design  coupling)  specifically  to  meet  the  critical  component 
requirements,  however  a  technology  which  will  expand  (Design  Parameter). 

•  A  methodology  of  delivering  just  the  interface  to  and  from  the  components, 
streamed  line  for  global  access,  performance  in  near-time  and  system  delivered  in 
less  than  a  year  time  (Critical  Key  Parameters). 

•  A  unique  approach  is  required,  which  resulted  in  a  new  way  for  successful 
Application  Integration  and  Deployment  of  Data  with  the  demands  specified. 

•  It  is  being  called  Point  Service  Enterprise  Architecture  (PSEA). 

•  Where  a  Point  Service  Enterprise  Architecture  links  an  enterprise’s  business 
architecture  with  its  existing  enterprise  systems  and  applications  utilizing  existing 
software  component  frameworks  that  can  be  applied  specifically  to  meet  a 
business  practice.  Point-to-Point  Application  Services. 

•  .NET,  J2EE,  SOA  Architectures,  ect  all  are  enabling  technologies,  it’s  the 
arrangement  of  the  frameworks  interfaced  to  existing  systems  with  interlacing 
services  over  a  business  process. 
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PSEA 


•  A  well-documented  process  architecture  is  critical,  with  a  precise 
logical  organization  of  information  pertaining  to  the  following 
elements: 

7i  Strategic  goals,  objectives,  and  strategies 
7i  Business  rules  and  measures 
7i  Information  requirements 
7i  Processes,  systems  and  applications 
7i  Relationships  between  architecture  elements 
7i  Technology  infrastructure 
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PSEA  High-Level 
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Conclusion 


•  (PSEA)  Point  services  allow  enterprises  to 
develop  applications  on  globally  distributed 
computing  platform  effectively. 

•  Modernization  Efforts,  bridging  Contractor  to 
Government: 

•  GLSN  (Global  Logistics  Support  Network) 

•  CLOE  (Common  Logistics  Operating 
Environment) 

•  VHMS  (Vehicle  Health  Management  Systems 
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Questions,  Other  Information 


•  Whitepaper  on  PSEA  is  available 

•  Proven  -  GD  Enterprise  and  Army 
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Summary 

Institutionalize  Best  Practices 
Improve  Program  Oversight 

Integrate 


Collaborate 


JOHNS  HOPKINS 

UNIVERSITY 


Systems  Engineering  of  Deployed 

Systems 

Bob  Finlayson 
Bryan  Herdlick 

The  Johns  Hopkins  University,  Whiting 
School  of  Engineering 


Purpose 


JOHNS  HOPKINS 

UNIVERSITY 


•  Understand  the  role  and  function  of  the  system  engineer  during  the 
operations  and  support  phase  of  a  system 

-  Understand  logistic  support  considerations  and  how  they  influence 
design,  manufacturing,  production  and  operations  decisions 

-  Identify  system  supportability  challenges  and  the  means  to  address 
them 

-  Develop  deployed  support  resource  requirements  for  system  life 

-  Master  the  ability  to  address  system  modifications  in  a  dynamic 
environment 


Challenge 


JOHNS  HOPKINS 

UNIVERSITY 


•  “The  operations  and  support  phase  of  the  system  life 
cycle  is  the  time  during  which  the  products  of  the  system 
development  and  production  phases  perform  the 
operational  functions  for  which  they  were  designed.  In 
theory,  the  tasks  of  systems  engineering  have  been 
completed.  In  practice,  however,  the  operation  of 
modern  complex  systems  is  never  without  incident.  ” 


Kossiakoff,  A.,  Sweet,  W.,  Systems  Engineering  Principles  and  Practice 


Course  Focus 


JOHNS  HOPKINS 

UNIVERSITY 


•  What  is  peculiar  about  this  aspect  of  the  lifecycle  &  related  SE  topics 
in  the  context  of  mature  /  deployed  /  legacy  systems? 

•  What  lessons  learned,  best  practices,  tools  should  the  systems 
engineer  be  familiar  with? 

•  What  are  the  risks  that  the  SE  should  watch  out  for? 

•  Are  there  rules  to  live  by? 

•  What  is  the  role  of  the  systems  engineer  in  context  of  deployed  / 
mature  /  legacy  systems? 


This  is  not  a  course  in  logistics  management,  but  the  systems 
engineer  must  have  a  thorough  understanding  of  the  logistics 
discipline  if  he  or  she  hopes  to  address  the  engineering  challenges  of 

deployed  systems 


Scope 


JOHNS  HOPKINS 
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•  System  under  design  -  usually  in  the  early  stages  of 
initial  design  or  during  the  design  of  deployed  system 
upgrades 

•  Operating  environment 

•  System  developer  and  manufacturer 

•  Manufacturer’s  supply  chain 

•  Logistics  elements  and  their  impact  on  systems 


Course  Flow 


JOHNS  HOPKINS 

UNIVERSITY 


System  Engineering  is  Part  of 
Project  Management 


JOHNS  HOPKINS 

UNIVERSITY 


Project  Management 


Project 

Planning 

&  Control 


Project  Guidelines 

-  Policy 

-  Goals 


Funds  Allocation 

-  Program  Commitments 

-  Subcontracts 


Resource 
Allocation 
-  Manpower 
_  _  I  .Facilities  _  J 


Task  Definition 

-  Plans 

-  Schedules 

-  Outputs 


!  Sponsor 
Interaction 

-  Management 

-  Technical 


System 

Engineering 


Technical  Guidelines 

-  Concept/Tech  Approach 

-  Objectives/Criteria 


System  Integration 

-  Tech.  Status 

-  System  Performance 

-  Interfaces 

-  System  Documentation 


Technical 

Coordination 

-  Design 

-  Tech.  Disciplines 

-  Subcontracts 


Acguisition  Logistics 


Manufactured  and  Production 


Operations  and  SuoDort 


Loaistic  elements 


Logistics  Management 

A  systems  approach 


JOHNS  HOPKINS 

UNIVERSITY 


KPP  -  Key  Performance  Parameter 
KSA  -  Key  System  Attribute 
SE  -  Support  Equipment 


Deployed  System  Design 


JOHNS  HOPKINS 

UNIVERSITY 


What  aspects  or  attributes  of  deployed 
systems  do  we  typically  worry  about? 

-  Reliability 

-  Maintainability 

-  Training 

-  Supply  support 

-  Health  and  status 

-  Safety  (Operational  Risk) 

-  Adaptability  |  . .  .  .  , 

K  1  How  do  we  account  for 

-  Upgradeable  these  in  the  design 

-  Disposability  phases,  during 

-  Cost  production  and  then 

again,  once  the  system  is 

deployed? 


Limitations/Constraints 


JOHNS  HOPKINS 

UNIVERSITY 


•  Analyses  limitations 

-  Availability  of  data  to  conduct 

-  Time  to  complete 

-  Resources 

•  Funding 

-  Deployed  phase  often  relegated  to  second  tier  status 

•  “Worry  about  it  later”  mentality 

•  Change  in  funding  source 

•  Lack  of  R&D  funds  in  deployed  phase 

•  System  design 

-  May  be  “frozen” 

•  Concept  of  operations  (CONOPS)  and  the  associated  tempo  are 
already  established 

-  Reluctance  to  alter  CONOPS  based  on  new  capability 


Manufacturing  &  Production 


JOHNS  HOPKINS 

UNIVERSITY 


Admin 


•  Instructor  -  Bryan  Herdlick 

•  Learning  Objectives 

•  Establish  an  understanding  of  fundamental 
manufacturing  &  production  processes 

•  Identify  SE  principles  and  activities  that  influence 
effectiveness  of  manufacturing  and  production 

•  Understand  the  responsibility  of  the  systems  engineer 
relative  to  manufacturing  &  production 

•  Preparation 

•3.1, 4.4(c),  5.2.2,  6.2.4  /  Chapter  7  /  TBD 

•  Homework  Problems 

•  3.1,  3.4,  TBD 


Lecture  Topics 

•  Production  as  a  system 

•  Producibility 

•  Designing  for  Manufacture 

•  Analysis  &  Metrics 

•  Facility  /  Utility 

•  Operational  Equipment  Effectiveness 
• FMECA 

•  Depot  Maintenance  &  Warranty  Repair 

•  Test 

•  Upgrades 

•  Foreign  Military  Sales 

•  Engineering  Disciplines  and  the  Systems  Engineer 


Take  Aways 

•  A  stable  process,  with  quantifiable  &  meaningful  metrics, 
active  monitoring  and  control  programs,  and  characteristic 
workforce  ‘ownership’  is  a  prerequisite  for  any  successful 
improvement  efforts. 

•  TBD 

•  TBD 

•  Example  :  JSF  Airframe  Affordability  Demonstration 


Depot  /  Warranty  Repair 


JOHNS  HOPKINS 

UNIVERSITY 


Facility  Design  &  Production  Optimization 


JOHNS  HOPKINS 

UNIVERSITY 


Lecture  Topics 


*  Goals  &  Benefits 

*  Cost  of  Quality 

*  Tools 

• FMECA 

•  Six  Sigma 

•  Lean  Manufacturing 

*  Industrial  Engineering 

*  Facility  design 

*  Manufacturing  process 

*  Role  of  SE  and  the  systems  engineer 


Admin 


Take  Aways 


•  Instructor  -  Bryan  Herdlick 

•  Learning  Objectives 

•  Identify  aspects  of  facility  design  and  the  production 
process  that  can  influence  efficiency 

•  Establish  a  basic  understanding  of  the  tools  available  to 
monitor  and  optimize  production  activities 

•  Understand  the  responsibility  of  the  systems  engineer 
relative  to  improving  manufacturing  &  production 
efficiency 

•  Preparation 

•  5.2.2,  6.2.4,  Chapter  7 

•  Homework  Problems 

•  6.10(a&c),  6.18,  Chapter  7:  1,  3,  5,  8,  11,  12,  27,  30,  32 


•  The  production  process,  including  the  associated  facilities, 
is  a  system  unto  itself  and  is  well  suited  to  the  application 
of  basic  systems  engineering  principles. 

•  Each  production  “batch”  or  “lot”  is  an  iteration  in  the 
collection  of  reliability  data  and  insight  into  opportunities 
for  enhanced  efficiency,  with  recommendations  for 
improvement  becoming  more  accurate  and  actionable 

•  The  systems  engineer  serves  a  vital  role  in  the  planning 
and  conduct  of  successful  production  activities  by  bridging 
multiple  engineering  disciplines  and  facilitating  cooperative 
process  &  design  improvement  efforts 


JOHNS  HOPKINS 

UNIVERSITY 


Production  Optimization 

(CONTINUOUS  PROCESS) 


•  Achieving  peak  effectiveness  through  continuous  efficiency 
enhancement 

-  Production  line  reliability  is  key 

-  Minimize  down-time 

•  Preventative  Maintenance  (PM) 

-  Reliability  Centered  Maintenance  (RCM)  approach 

»  PM  only  when  justified  ( reliability  data,  physics-of-failure,  etc.) 

•  Continuous  production  (i.e.  no  “breaks”  in  production  runs) 

•  Maintainability  features 

-  Involve  operators  in  ongoing  process  analysis  and  improvement 

•  Responsibility  for  process  “escapes” 

•  Responsibility  for  initial  troubleshooting 

•  Understanding  of  cause-effect  relationships  =  “ownership” 


Each  production  “batch”  or  “lot”  is  an  iteration  in  the  collection  of  reliability 
data  and  insight  into  opportunities  for  enhanced  efficiency,  with 
recommendations  for  improvement  becoming  more  accurate  and  actionable 

as  each  cycle  is  completed. 


Major  System  Upgrade  Challenges 


JOHNS  HOPKINS 

UNIVERSITY 


•  Upgrades  are  often  pursued  without  due  diligence  in  one 
or  more  of  the  following  areas: 

-  Requirements  refinement  &  validation 

-  Supportability  Analysis 

-  Configuration  Management 

-  Accurate  assessment  of 

•  Design  /  integration  challenges 

•  Technology  maturity 

In  addition  to  ensuring  that  a  system  upgrade  satisfies  requirements  for 
corrective  action  or  performance  enhancement,  the  systems  engineer  is  also 
responsible  for  maintaining  or  improving  the  suitability  of  the  fielded  system 
-  including  both  supportability  and  lifecycle  affordability 


The  Systems  Engineer 

(In  the  context  of  system  upgrades) 


JOHNS  HOPKINS 

UNIVERSITY 


Controls  &  Constraints: 

-  Funding 

-  Schedule  drivers 

-  Maturity  of  technology 

-  New  statute  /  policy 


Conducting  a  Logistics 
Supportability  Analysis 
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•  An  LSA  can  aid  in: 

-  Initial  establishment  of  supportability  requirements  during  conceptual 
design 

-  Early  establishment  of  supportability  design-to  criteria 

•  Definition  of  system  operational  requirements 

•  Maintenance  and  support  concept 

•  Identification  and  prioritization  of  technical  performance  measures 

•  Performance  of  functional  analysis 

•  Allocation  of  requirements 

-  Synthesis,  analysis  and  design  optimization  effort  through  trade  studies 

•  Alternative  repair  policies 

•  Reliability  and  maintainability  characteristics 

•  Commercial-off-the-shelf  implementation 

-  Evaluation  of  a  given  design  configuration 

-  Assessment  of  an  operating  system’s  effectiveness  and  supportability  in 
its  intended  environment 


LSA  Tools 


JOHNS  HOPKINS 

UNIVERSITY 


•  Life  Cycle  Cost  Analysis  (LCCA)  (Session  13) 

-  Total  cost  of  the  system  and  its  supporting  activities  throughout  the  life  of  the 
system 

•  Failure  Mode,  Effects  and  Criticality  Analysis  (FMECA)  (Session  6) 

-  Identification  of  potential  system  and/or  process  failures,  the  expected  mode  of 
failure  and  causes,  failure  effects  and  mechanisms,  anticipated  frequency, 
criticality  and  the  steps  required  for  compensation 

•  Fault  Tree  Analysis  (FTA) 

-  Deductive  approach  involving  graphical  enumeration  of  different  ways  a  failure 
can  occur  and  its  probability  of  occurrence 

•  Maintenance  Task  Analysis  (MTA)  (Session  7) 

-  Maintenance  functions  to  be  allocated  to  a  human 

•  Reliability  Centered  Maintenance  (RCM)  (Session  3) 

-  Best  overall  approach  for  preventative  maintenance 

•  Level-of-Repair  Analysis  (LORA)  (Session  7) 

-  Maintenance  policies  in  terms  of  level  of  repair 

•  Evaluation  of  Design  Alternatives  (Analysis  of  Alternatives  (AoA)) 

-  Assess  design  configurations  using  multiple  criteria 


Addressing  Reliability 


JOHNS  HOPKINS 

UNIVERSITY 


•  It’s  one  thing  to  teach  reliability  theory,  it  is 
another  to  apply  it  in  the  proper  manner 

•  Do  you  truly  understand  the  problem  at  hand? 

-  Environment,  requirements,  CONOPS 

•  Have  you  set  the  boundary  conditions? 

-  Assumptions,  limitations 

•  Have  you  correctly  assumed  equilibrium? 

-  Modeling 

•  Can  you  solve  for  the  unknowns? 

-  Design  and  verification 
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This  is  a  typical  behavioral 
model  for  an  organic  and 
inorganic  system  -  looks  fairly 
benign,  but  there  is  much  more 
to  the  curve  than  depicted  here  - 
randomness,  environmental 
effects,  catastrophic  events,  etc. 
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FMECA  Approach 
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Analyze  Failure  Mode  Criticality 


Overall  Maintenance 
Conceptualization 


JOHNS  HOPKINS 

UNIVERSITY 


•  Why  -  reusable  or  disposable 

•  Who  -  personnel  requirements  and  limitations 

•  What  -  type  of  maintenance  to  be  performed  (electronic, 
software,  structural,  mechanical) 

•  Where  -  field  environment  or  designated  repair  facility 

•  When  -  Planned  versus  unscheduled 

•  How  -  appropriate  level  of  maintenance 


Maintenance  Planning: 
Environmental 


JOHNS  HOPKINS 

UNIVERSITY 


•  Location 

-  Constraints 

•  Space,  accessibility  to  the  system 

-  External  factors 

•  Weather,  contaminants 

•  Supply  chain 

-  Provide  the  necessary  support  infrastructure  to  conduct 
maintenance  actions 

•  Number  of  personnel  available 

-  Limited  detachment 

-  Provisioning 

•  Support  equipment 

-  Weight,  volume,  fragility 


Spares  Hypothesis  Testing 


JOHNS  HOPKINS 
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•  Following  deployment  of  a  system  (and  throughout  the 
lifecycle)  the  requirements  for  spares  and  parts  must  be 
reevaluated  based  on... 

-  Actual  system  performance  /  reliability  /  availability 

-  Changes  in  the  operating  and  programmatic  environments 

-  Changes  in  the  maintenance  concept  or  production 


Planning  for  adequate  spares  and  repair  parts  (including  subassemblies)  is 
based  upon  assumptions  and  predictions  that  must  be  continuously 
reviewed  in  light  of  post-deployment  system  performance  and  maintenance  / 

repair  activities 


Spares  Calculation 


JOHNS  HOPKINS 
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•  What  goes  into  predicting  spares  requirements? 

-  Failure  rates 

•  Individual  parts 

•  Subassemblies 

•  Composite  system 

-  Spares  procurement  &  stock  intervals 

•  Predicated  on  one-for-one  replacement  maintenance 
concept 

-  Mission  duration 

-  Number  of  systems  in  service  (available  to  satisfy 
mission) 


•  K  AT  =  Translation  factor 

-  K  =  Number  of  Parts  (per  assembly  under 
consideration) 

-  A  =  Part  failure  rate 

-  T  =  Interval  for  procurement  of  stock  /  spares 


JOHNS  HOPKINS 

UNIVERSITY 


PHS&T: 

Implementing  a  Supply  Chain 


Corporate  Strategy 


What  are  the  strategic  objectives  with  regard  to 
logistics? 

-  In-house  transportation  management 

-  Investment  in  automated  systems 

-  Customer  liaison  policy 

-  Warehousing  and  inventory  management 

-  Corporate  reach  (global?) 


Metrics 


Are  adequate  measures  in  place  to  assess  progress? 

-  Requirements  articulated 

-  Supply  chain  modeling 

-  Design  in  place 

-  Functional  flows  understood 

-  Trades  analyses  identified 


Execution 


Is  the  infrastructure  in  place  to  execute,  monitor  and 
control  the  process? 

-  Personnel 

-  Tools 

-  Visibility 

-  Quality  Management 

-  Risk  forecasting 


Typical  Training  Requirements 


JOHNS  HOPKINS 
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•  System  design 

-  Human  system  interface,  operational  environment 


•  Training  facilities 
-  Location,  size 


•  Throughput 

-  Number  of  training  events  per  day 

•  Data  capture  and  recording  -  feedback 


•  Task  Complexity 


•  SS  environmental  predictability 

-  Numerically  controlled  machine  maintenance 

-  Driver’s  education 


Predictability 


Training  System  Interaction 
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Disposal 


JOHNS  HOPKINS 
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Admin 


•  Instructor  -  Bryan  Herdlick 

•  Learning  Objectives 

•  Discuss  disposal  considerations  associated  with  each 
phase  of  a  system’s  lifecycle 

•  Identify  SE  principles  and  activities  that  can  assist  in 
avoiding  or  managing  disposal  risks  and  cost 

•  Understand  the  role  and  responsibilities  of  the  systems 
engineer  relative  to  system  disposal 

•  Preparation 

•  See  text  reference  list 

•  Homework  Problems 

•  Problems:  8.23,  8.27,  8.28,  8.29,  8.30,  8.31 


Lecture  Topics 

•  Disposal  at  retirement 

•  Disposal  during  the  lifecycle 

•  Environmental  considerations  /  impact  statements 

•  Designing  for  disposal 

-  Disposal  considerations  during  mod  /  upgrades 

-  Ties  to  maintenance  plan  decision  tree 

•  The  “Zero  Waste”  ideal 

-  Reuse  /  reclaim  options 

-  Salvage  operations 

-  Cost,  benefits  and  examples 


Take-Aways 

•  Disposal,  as  an  activity,  is  not  relegated  to  system 
retirement  and  the  end  of  the  lifecycle 

•  Disposal,  as  commonly  defined  (e.g.  dumping,  discarding, 
throwing  away),  should  be  considered  the  least  efficient 
and  least  desirable  alternative  for  the  processing  of 
residual  /  waste  materials 

•  A  system  successfully  “designed  for  disposal”  would 
incorporate  extensive  use  of  alternatives  to  disposal  such 
as  salvage,  recycling,  reuse 

•  Exceptionally  high  quality  products  /  systems  evidence 
longevity  that  can  reduce  retirement  waste  (but  mid-life 
maintenance  waste  still  exists) 


“Disposal”  in  the  Design  Checklist 
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•  Has  disposability  been  evaluated  during  design? 

-  Is  recycling  or  re-use  of  components  an  alternative? 

-  Is  decomposition  /  disassembly  an  alternative  (requirement)? 

-  Are  additional  logistic  support  resources  required? 

•  Have  disposal  procedures  been  identified  /  prepared? 

-  Are  methods  &  results  consistent  with  environmental,  safety, 
political  and  social  requirements 

-  Are  the  methods  economically  feasible? 


Life  Cycle  Cost  Analysis 


JOHNS  HOPKINS 

UNIVERSITY 


•  LCCA  is  presented  at  the  end 
of  the  course 

-  Necessary  to  understand  the 
other  elements  in  order  to 
conduct  a  LCCA 

-  Serves  as  a  review  of  the 
material 

•  Can  only  address  certain 
aspects  of  the  LCCA 

-  Too  encompassing  to  cover  it 
completely 

-  SE  contributes,  but  typically 
does  not  conduct  the  LCCA 
itself 

•  However,  a  primary  driver  not 
only  for  decision  making,  but 
for  keeping  O&S  activities 
under  control 


Operation  ROTE 


Deployed  Systems  Engineering 

Risks 


JOHNS  HOPKINS 

UNIVERSITY 


•  Confusing  performance  requirements  with  supportability 
requirements 

-  Can’t  have  one  without  the  other,  but  there  is  a  tension  between  them 
many  cases 

•  Incomplete  understanding  of  requirements  and  their  allocation  to 
system  functions 

•  Assigning  the  wrong  measures  (and  the  respective  values)  to  the 
system  evaluation  process 

•  Addressing  3  or  4  of  the  primary  logistic  elements  (e.g., 
maintenance,  personnel),  while  ignoring  the  rest 

•  Designing  for  O&S  at  the  component  level  without  regard  for  the 
system  and  its  internal  and  external  interactions 


Deployed  Systems  Opportunities 


JOHNS  HOPKINS 
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•  Good  systems  engineering  is  necessary  in  the  O&S  phases 

-  Success  is  not  in  deploying  a  system,  but  in  the  system  performing  its 
intended  role  effectively  and  efficiently  for  its  entire  duration 

•  A  good  SE  approach  will  reveal  risks  and  challenges  that  often  go 
unseen  until  a  system  is  too  far  along  in  the  design  process 

-  Costly  upgrades 

-  Performance  degradation 

-  Excessive  schedule  delays 

•  Understanding  and  applying  a  disciplined  technical  approach  is 
necessary  for  all  phases  of  the  life  cycle 

-  Computers  can  crunch  numbers,  but  they  cannot  build  a  credible  model 

•  Intuition,  discipline,  accurate  assumptions 

-  Technical  leadership  that  encompasses  many  disciplines 


BAE  SYSTEMS 


Welcome  to  BAE  Systems 


Mobility  &  Protection  Systems,  Sterling  Heights,  Ml  -  October  2008 


03-0ctober-2008 
Property  Of  BAE  Systems  M&PS 
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BAE  SYSTEMS 


Systems  Engineering  Approach 


Systems  Engineering  in  New  Vehicle 
Development 

FTTS  (Future  Tactical  Truck  Systems) 

Customer:  US  TACOM  National  Automotive  Center  (NAC),  Warren  Mi 

Walter  J.  Budd 
Chief  Engineer 
BAE  M&PS 
October  2008 


03-0ctober-2008 
Property  Of  BAE  Systems  M&PS 


BAE  SYSTEMS 


Vehicle  Design  Process 


MSV  -  Maneuver  Sustainment  Vehicle 

■18  Month  Project,  Design,  Build,  Qualify  New  Vehicle 

■Systems  Engineering  Approach 

■  Requirements  Analysis 

■Performance  Parameters  Linked  Into  Models 


03-0ctober-2008 
Property  Of  BAE  Systems  M&PS 


BAE  SYSTEMS 


Requirements  Analysis 


•  Process  Began  With  Customer  Supplied  92 
Page  Performance  Requirement  Document 

•  Our  Engineers  Developed  and  Tracked  408 
Given  and  Derived  Requirements 


Parameters: 

408 

Parameter  Index 

Parameter  Name 

Specification  Val 

Specification 

Priori 

Level 

Units 

Operator 

Thre; 

0 

eJ  eJ  eJ 

i_ s 

E 

_ 

3 

.1  -  Mission  Profile 

K-Key 

MOBILE IMPROVED HARD 

% 

= 

53 

3 

.1  -  Mission  Profile 

K-  Key 

MOBILE  IMPROVED  GRAVEL 

% 

= 

7. 

3 

.1  -  Mission  Profile 

K-Key 

MOBILE UNIMPROVED CC 

% 

= 

39 

hr 

.1  -.1  -Li 

D  -  Design 

CURB  WEIGHT 

lbs 

<= 

282C 

3 

.1  -.1  - .2 

D  -  Design 

GVW 

lbs 

<= 

575S 

nr 

.1-1.1  -\.2 

D  -  Design 

SOLDIER NUMBER 

= 

2.( 

1  3 

.1-|.1  -[.2 

D  -  Design 

MASS  SOLDIER 

lbs 

= 

342 

nr 

.1  -.1  -  .3 

D  -  Design 

GCW 

lbs 

<= 

10101 

hr 

.1  -.1  -  .3 

D  -  Design 

MASS  GT _ 

lbs 

<= 

202C 

►i  \  Specification  Parameters  / 
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BAE  SYSTEMS 


Requirements  Verification 
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Propulsion  System 


Propulsion  Modules 
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Propulsion:  Engine/E-Motor 


BAE  SYSTEMS 


Power  Generation 

Hybrid  System 


CAT  C9 
1600  RPM 
1850  N.m 
450  Hp 


UQM 

1900  RPM 
600  N.m 
160  Hp 


Combined  Peak  Torque  Combined  Peak  Power 

2446  N-m  (1804  ft-lbf)  @  1600  RPM  610  HP  (455  kW)  @  2300  RPM 
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Propulsion:  Batteries 


Four,  45A*h 
NiMH  Batteries 
Used  To  Support 
The  Hybrid 
Power 

Requirements 


MANUFACTURER 

COBASYS 

TYPE 

NiMH 

MODEL 

4500  SERIES 

VOLTAGE 

V 

336 

CAPACITY 

Ah 

45 

COOLING 

LIQUID,  INTEGRATED 

DRY  WEIGHT 

Kg 

330 

No.  of  BATTERIES 

28 

DIMENSIONS: 

LxWxH 

MM 

1900x600x310 

BAE  SYSTEMS 


Battery  Pack 
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Propulsion:  Axles 


Custom  Designed  Independent 
Suspension  Axles 


■  Independent  Suspension  SLA 

■  Axel  Differential  Ratio:  2.077 

■  Wheel  Hub  Planetary  Ratio:  3.55 

■  Hydraulic  Disc  Brakes  -  ABS 


BAE  SYSTEMS 
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BAE  SYSTEMS 


Frame  Design/  Stress  Analysis 


All-New  Frame 
Was  Required 


Inputs  from: 

•  Automotive 
Loads 

•  13  Ton  Load 
Carry 

•  Lift/Unload  13 
Ton  Cargo 
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BAE  SYSTEMS 


Material  Handling  Equipment 


Material  Handling  Equipment 
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Material  Handling  Equipment 


BAE  SYSTEMS 


Material  Handling  Equipment 
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Material  Handling  Equipment 


BAE  SYSTEMS 


Get  supplies  to  the  soldiers  as  quickly  and  as 

safely  as  possible 
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BAE  SYSTEMS 


Material  Handling  Equipment 


•  Load/Unload  Cargo 


•  Load/Unload  Trailer 


•  Load/Unload  ISO  Containers 


03-0ctober-2008 
Property  Of  BAE  Systems  M&PS 
14 


Mobility  Models 


Challenge 

•  Create  multibody 
simulation  that  represents 
several  truck  and 
suspension  variants 

•  Different  suspension 
designs  (not  just 
parameter  values) 

•  Make  it  easy  to  run 
different  trucks  on  all 
possible  roads  and 
obstacles 


Mobility  Models 


Model  as  a  series 
of  rigid  bodies  with 
joints  and  force 
elements 

Tire  forces 
modeled  for  both 
hard  and  soft 
surfaces 

Driving  scenarios 
to  test  limit 
handling  in  loaded 
condition 


BAE  SYSTEMS 


Mobility  Models 


■Lane  change 
stability  test 

■Predict  handling 
stability  and  peak 
roll  and  lateral 
accelerations 
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Mobility  Models 


■Predict  roll,  sliding 
and  dynamic  loads 

■Verify  safe 
operating  limit  for 
field  tests 

■Avoid  dangerous 
tests  that  could 
endanger  drivers 
and  prototype 
equipment 


BAE  SYSTEMS 
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BAE  SYSTEMS 


Results 
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Results 


BAE  SYSTEMS 


MSV:  Measured  and 
tested  to  the  limits 
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BAE  SYSTEMS 


Results 


MSV 

Core 

Team 
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BAE  SYSTEMS 


Conclusion 


■  Lessons  Learned 

•  Value  Of  The  Systems  Engineering  Process 

•  Importance  Of  Model  Validation 

•  Benefits  To  BAE  Systems 

•  Benefits  To  The  Customer 
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Conclusion 


BAE  SYSTEMS 


% 
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I  ntroduction  -  Requirements  Decomposition 


NORTHROP  GRUMMAN 


•  What  is  typically  done  when  requirements  are  decomposed 
or  flowed  down? 

-  Split  up  one  function  into  smaller  ones  and  allocate  them  to  various 
components,  so  that  when  each  component  performs  its  function,  the 
entire  function  will  be  completed. 

-  Performance  requirements  on  timeline/ speed/  accuracy,  etc.  are  divided  up 
similarly. 

•  Requirements  must  be  verifiable  or  testable 

-  Otherwise,  one  cannot  tell  if  requirements  are  implemented  correctly. 

•  Requirements  need  to  be  sold  off 
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Challenge  1  -  Timing  (1  of  5) 


NORTHROP  GRUMMAN 


•  Example  1  -  "classic"  decomposition 

-  The  system  shall  complete  the  task  within  10  sec 

•  A  shall  complete  its  task  within  3  sec. 

•  B  shall  complete  its  task  within  4  sec. 

•  C  shall  complete  its  task  within  2  sec. 

•  Margin  -  1  sec 

•  At  a  minimum 

-  The  start  &  end  points  at  each  component  must  be  measurable 

-  Need  a  well-defined  boundary  between  the  2  components 

•  Won't  work  if  B  is  an  embedded  library 

•  Architecture  &  design  are  important 

-  If  components  are  not  divided  up  logically,  there  will  be  issues 
with  verifying  requirements 
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Challenge  1  -  Timing  (2  of  5) 


NORTHROP  GRUMMAN 


•  The  classic  decomposition  works  just  fine  if  the  10  seconds 
is  a  generous  number,  and  each  component's  worst  case 
timing  is  within  the  allocation. 


Component 

Allocation 

Typical 

Worst  case 

A 

3 

2 

2.5 

B 

4 

2 

3 

C 

2 

1 

1.5 

Total 

9 

5 

7 

4 
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Challenge  1  -  Timing  (3  of  5) 


NORTHROP  GRUMMAN 


■  When  the  timing  is  tight 


Component 

Allocation 

Typical 

Worst  case 

A 

3 

2.5 

4 

B 

4 

4 

5 

C 

2 

1.9 

3 

Total 

9 

8.4 

12 

5 


10/30/2008  10:04  AM 


UNCLASSIFIED 


Challenge  1  -  Timing  (4  of  5) 


NORTHROP  GRUMMAN 


f  I  n  the  case  when  the  time  allowed  is  very  tight,  requiring 
each  component  to  individually  satisfy  the  timing  allocation 
may  be  a  problem. 


•  By  measuring  the  time  for  the  whole  system,  there  is  a 
much  better  chance  of  passing  the  requirement. 


*  May  need  customer  agreement  to  bypass  the  tests  for 
individual  components  and  test  the  system. 
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Challenge  1  -  Timing  (5  of  5) 


NORTHROP  GRUMMAN 


X  varies  with 

-  Atmospheric  absorption 

-  Background 

-  Position  w.r.t.  scan  pa 
or  starer  step  pattern 


Y  varies  with 
-  Photon  ->  exceedance 
(background/  clutter) 


Z  varies  with 

-  State  of  OS 

-  Load  of  server 


Time  =  X  +  Y  +  Z 
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Challenge  2  -  High  Level  of  Accuracy  (1  of  4) 


NORTHROP  GRUMMAN 


•  Example  2A 

a)  The  system  shall  do  something  99%  of  the  time. 

b)  The  system  shall  do  something  99.99%  of  the  time. 

•  Mathematical  distinction 

a)  To  be  able  to  distinguish  99%  from  98%, 
at  least  100  test  cases  must  be  run 

b)  To  be  able  to  distinguish  99.99%  from  99.98%, 
at  least  10,000  test  cases  must  be  run 
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Challenge  2  -  High  Level  of  Accuracy  (2  of  4) 


NORTHROP  GRUMMAN 


•  I  n  order  to  obtain  results  that  are  statistically  meaningful, 
larger  samples  are  needed,  so  the  no.  of  test  cases 
needed  will  be  even  bigger 


•  Cost  for  running  a  test  can  be  expensive 


•  Therefore,  the  consequence  of  higher  accuracy  must  be 
understood 
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Challenge  2  -  High  Level  of  Accuracy  (3  of  4) 


NORTHROP  GRUMMAN 


•  Example  2B 

How  does  the  higher  accuracy  creep  in  at  decomposition? 

The  system  shall  do  something  0.99  of  the  time 

•  The  accuracy  is  divided  up  as  follows: 

A  shall  do  its  task  0.9995  of  the  time 
B  shall  do  its  task  0.9905  of  the  time 

•  So  that  0.9995  X  0.9905  >  0.99 
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Challenge  2  -  High  Level  of  Accuracy  (4  of  4) 


NORTHROP  GRUMMAN 


•  Therefore,  the  consequence  &  trade-off  of  decomposing 
requirements  this  way  must  be  understood 

I  s  the  higher  accuracy  for  each  component  needed? 

With  the  much  larger  number  of  test  runs  for  each  component,  is  the 
system  gaining  a  higher  accuracy  as  a  result? 
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Challenge  3  -  Non-normal  Data  (1  of  5) 


NORTHROP  GRUMMAN 


•  Statistical  distinction 

I  n  order  to  obtain  results  that  are  statistically  meaningful,  larger 
samples  are  needed.  Furthermore,  the  more  the  data  does  not  follow 
the  normal  distribution,  the  bigger  the  sample  size  should  be. 


The  distribution  of  the  data  measured  is  usually  not  well  understood. 
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Challenge  3  -  Non- normal  Data  (2  of  5) 


NORTHROP  GRUMMAN 


•  An  example  of  binomial  distribution 

Probability  of  success  of  each  trial  is  0.5  or  0.95 
Total  no.  of  trial  is  100 


Binomial  Distribution 


Prob  ofSuccess  =  0.5 


—a —  Prob  of  Success  =  0.95 
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Challenge  3  -  Non- normal  Data  (3  of  5) 


NORTHROP  GRUMMAN 


|  f  0. 

2  is  o. 
12  o. 

h  o. 


-o. 


Cumulative  Binomial  Distribution 


20 


40 


60 


80 


No.  of  Successes 


T 

K 

f 

r 

-m- 


m 


■4—  Prob  of  Success  =  0.5  — 4—  Prob  of  Success  =  0.95 
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Challenge  3  -  Non-normal  Data  (4  of  5) 


NORTHROP  GRUMMAN 


•  Example  3 

The  system  shall  do  something  1/ month 


•  I  n  both  Examples  2  &  3,  the  probability  of  success  is  close 
to  zero  or  one,  i.e.  the  data  is  far  from  a  normal 
distribution. 

Assuming  a  minimum  of  10  samples  are  needed 

99.99%  =>  need  to  run  test  case 

(10,000  X  10)  =  100,000  times 

1/ month  =>  need  to  run  test  for  10  months 

*  If  the  requirement  is  on  the  order  of  1/year,  will  the  program  ever  have 
enough  time  to  test  the  requirement? 
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Challenge  3  -  Non- normal  Data  (5  of  5) 


NORTHROP  GRUMMAN 


•  If  there  are  not  sufficient  resources  to  run  the  large  no.  of 
tests  needed 

If  the  requirement  says  >=  0.95 

•  If  values  measured  are 
0.945,  0.955  &  0.951, 

is  it  conclusive  that  the  system  meets  the  requirement? 

•  Several  possible  solutions  to  this  situation 

Build  a  better  system 

Run  more  test  cases 

Buyer  is  willing  to  accept  the  risk 

•  The  smaller  the  no.  of  samples, 
the  higher  the  risk 
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Challenge  4  -  Simulated  Data  (1  of  2) 


NORTHROP  GRUMMAN 


•  Limitation  of  Simulated  Data 

-  I  n  order  to  verify  requirements,  simulated  data  will  often  be  needed. 
However,  the  limitations  of  simulated  data  (or  the  models  from  which  the 
data  is  created)  must  be  understood 

•  The  physical  system  that  needs  to  be  modeled/ simulated  is  often  not 
well  understood  to  the  level  of  precision  required. 

E.g. 

How  well  do  we  think  we  can  simulate  the  weather  data  for  the  next 
10  days? 

How  well  can  we  simulate  a  coin  toss  &  to  what  level  of  fidelity? 

•  I  n  Example  2,  the  simulated  data  for  testing  99.99%  needs  to  be 
many  times  more  accurate  than  the  data  for  testing  99%.  Therefore, 
the  models  also  need  to  be  much  more  accurate. 
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Challenge  4  -  Simulated  Data  (2  of  2) 


NORTHROP  GRUMMAN 


•  Program  Decision 

-  The  program  needs  to  decide  how  much  resources  should  be  spent  on 
modeling  and  simulation 

•  Is  it  worth  spending  a  lot  of  resources  to  improve  the  simulated  data? 

•  An  alternative  to  spending  resources  on  simulation  is  to  conduct 
verification  after  the  system  is  fielded.  This  is  also  a  decision  that  the 
program  needs  to  make. 
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Recommendations  ( 1  of  2) 


NORTHROP  GRUMMAN 


•  Never  write  requirements  that  are  impossible  to  test  or  cost 
a  lot  to  test 

•  Get  involved  as  early  as  possible 

-  Try  to  influence  upper  level  requirements  as  early  as  possible,  so  that  you 
wont  have  bad  parent  or  grandparent  requirements 

-  Participate  in  other  systems  engineering  activities,  such  as  architecture 
development,  and  look  out  for  potential  problems 

-  The  earlier  a  problem  is  discovered,  the  less  expensive  it  is  to  fix  the 
problem 
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Recommendations  (2  of  2) 


NORTHROP  GRUMMAN 


•  I  f  you  are  stuck  with  them,  need  to  work  with  customer  to 
find  a  way  out,  e.g. 

-  Help  customer  to  understand  the 

•  Problem 

•  Alternatives 

•  Cost  vs  benefit  of  each  alternative 

-  Get  customer  agreement  on  testing  the  system  without  testing  individual 
components 

•  Example  1  &  2B 

-  Get  customer  agreement  on  a  lower  level  of  accuracy 

•  Example  2A 

-  Convince  customer  to  accept  higher  risk 

•  Example  3 

-  Get  customer  agreement  on  testing  the  system  after  it  is  fielded 

•  Example  4 
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Conclusion 


NORTHROP  GRUMMAN 


•  Systems/ Requirements  engineers  need  to 

-  Know  how  to  write/develop/decompose  requirements,  and  also  understand 
the  impact  of  requirements  written  in  various  manners 

-  Understand  how  the  system  works 

-  Be  proactive  and  get  involved  as  early  as  possible 

-  Be  involved  in  higher  level  requirements,  architecture,  etc. 

-  I  nclude  the  effort  in  the  BOE 

-  Work  with  customers 


•  Contact  I  nfo 

-  Eliza. Siu  @  ngc.com 

-  626-812-1013 
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Enabling  Systems  Engineering  with  an  Integrated 
Approach  to  Knowledge  Discovery  and 
Architecture  F  ramework 

Michael  R.  Collins 
Advantage  Development,  Inc. 

October,  2008 


925  Bassett  Road,  Suite  A 
P.  O.  Box  45154,  Westlake,  OH  44145 


•  Engineering  employs  analysis  of  function  to  iteratively  decompose 
and  separate  a  primarily  functional  representation  of  a  whole  into 
representations  of  economically  producible  components  that  can  be 
assembled  to  construct  the  functional  whole. 

♦  Big  implication  here!  Engineering  requires  an  “initial  point”  -  a 
representation  of  the  whole  —  to  be  successful! 

Engineering  does  not  work  without  an  initial  point!! 

•  We  refer  to  this  “initial  point”  as: 


Engineerible  Requirements 


The  set  of  engineering  requirements  necessary  and  sufficient  to  initiate 
the  successful  engineering  and  production  of  a  system 


Brad  Mercer,  MITRE,  Chief  Architect  Maritime  IT  and  Engineering 


©2008 
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Architecting  and  Engineering 
Different  Sides  of  the  Same  Coin 


•  Architecting  employs  synthesis  of  form  to  iteratively  compose 
separate  elements  to  form  a  coherent  whole,  or  a  representation  of  a 
coherent  whole,  that  can  serve  as  an  “initial  point”  for  system 
development. 

•  Architecting  synthesizes  this  “initial  point”  from  the  collective  vision, 
goals,  constraints,  and  other  needs  of  the  stakeholders  in  the  to-be- 
developed  system  —  converting  conflicting  stakeholder  demands  into 
a  conceptualized  whole  that  maximizes  the  satisfaction  of  each 
stakeholder. 

•  From  the  point  of  view  of  architecting,  we  refer  to  this  “engineering 
initial  point”  as  an: 

Architecture  Specification 

An  architecture  description  to  which  all  system  implementations  must 
adhere;  and  a  set  of  principles,  practices,  and  constraints  guiding 
implementation,  operation,  and  evolution  of  the  developed  system 
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Brad  Mercer,  MITRE,  Chief  Architect  Maritime  IT  and  Engineering 


Architecting  and  Engineering 
Different  Sides  of  the  Same  Coin 


collective  vision,  goals,  constraints, 
and  other  needs  of  the  stakeholders 


architecture  specification 


iteratively  compose 
separate  elements  to 
form  a  coherent  whole 


engineerible  requirements 


Analysis 
of  Function 

Engineering 


representations  of  economica™ 
producible  components  that  can  be 
assembled  to  construct  the  functional  whole 


iteratively  decompose  and 
separate  a  primarily  functional 
representation  of  a  whole 
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Brad  Mercer,  MITRE,  Chief  Architect  Maritime  IT  and  Engineering 
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Root  Process  Problem 


•  Complexity  of  data  elements  is  overwhelming 

•  Difficult  to  support  the  book-keeping  management 
of  all  of  the  data  elements  and  their  relationships 
across  all  the  echelons  of  the  Enterprise 

♦  ‘Structuring’  complexity 

♦  ‘Echelon  integration  and  enterprise  description’- 
everything  is  a  part  of  a  larger  system 

♦  Persistent,  iterative,  and  evolutionary  incorporation  in  a 
knowledge  and  reuse  environment 
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Discovery 


•  Every  object,  relationship  and  aggregation  of  objects 
in  the  knowledge  metamodel  is  in  documents,  the 
universe  of  textual  models 

•  Discovery  is  about  finding  the  objects,  relationships, 
aggregations  and  descriptions  of  each  of  these  in  the 
authoritative  and  original  data  sources 

•  Integration  is  about  using  Discovery  to  build  and 
describe  the  Architecture  using  an  architecture 
meta-model 
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Concept,  Themes,  and  Description 


•  A  concept,  or  theme,  is  the  encapsulation  of  a 
pattern  that  is  identified  as  a  gestalt:  a  persistent 
and  unique  ‘signature’ 

•  Documents  are  textual  patterns 

•  Models  are  labeled,  structured  patterns 

•  Labels  are  knowledge  anchors  to  concepts  and 
themes 

•  Knowledge  is  pattern  recognition,  association  and 
application  in  integrated  textual  and  model  gestalts 


©2008 

925  Bassett  Road,  Suite  A 
P.  O.  Box  45154,  Westlake,  OH  44145 


8 


DISCOVERING  CONCEPTS  /  THEMES 


•  ‘Information'  can  be  treated  as  quantifiable 
symbols  in  communications 

•  Natural  language  has  a  high  degree  of  unessential 
content,  the  less  frequently  a  unit  of  communication 
occurs,  the  more  information  it  conveys 

•  Information  objects  extracted  from  Natural 
Language  text  form  a  index  unique  to  that  concept 

•  The  architecture  metamodel  is  the  syntactic  of  the 
knowledge  pattern  and  is  semantically  rigorous 

•  Information  objects  cluster  based  upon  an  inference 
relationship  measuring  semantic  completeness 
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Example  Cluster  Picture  from 
the  Cartia:  ThemeScape  Web 
Site 


Cluster  the  indexes  of  the 
information  objects  using 
Statistical  Inference! 


Define  the  information 
objects  and  index  them 


THE  UNIVERSE  OF  DOCUMENT 


ATION 

OBJECTS  THAT  DESCRIBE  THE  IMPLEMENTATION  BASELINE 
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Cluster  and  Relationship  Visualizations 


Pictures  from  Battelle,  PNNL  Starlight  Web 
Site  http://starlight.pnl.gov/ 

Cluster  and  integrate  using  the  architecture  meta  model 
Visualization  can  take  many  forms  presenting  many  perspectives. 

Tracing  of  the  models  back  to  the  authoritative  and  original  data  sources. 
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DISCOVERY  PROCESS 


Discovery  works  best  when  it  has  a  pre-existent  form  upon 


which  it  can  operate. 
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ARCHITECTURE  &  DISCOVERY 


•  Architecture  is  the  description  of  the  intrinsic  relationships, 
characteristics  and  behaviors  of  the  system  under  study 

♦  All  systems  have  an  architecture  —  intentionally  architected  or  not  — 
and  that  architecture  is  a  primary  determinant  of  the  system’s 
behavior.  Brad  Mercer,  MITRE  Principal  Architect 

♦  Architecture  is  the  model  in  Modeling  and  Simulation  and  a  rigorous 
and  well-constructed  model  can  be  executed 

•  Discovery:  the  process  for  identifying  the  conceptual 
syntactic  of  architecture  and  the  rich  semantics 

•  Present  architecture  efforts  are  neither  semantically 
complete  nor  rich:  they  contain  a  series  of  model  artifacts 
(products)  built  and  limited  to  “ labeled ”  components  and 
relationships;  it  has  no  processes,  only  product  templates 
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Taxonomies  of  Primitives 


•  Indexing  and  clustering  builds  initial  identification 
and  organization  of  labeled  themes  and  concepts 

•  Clusters  are  labeled  taxonomical  elements 

•  Rich  taxonomies  can  be  developed  from  clusters 

♦  Structured  and  organized  categorization  of  information 

♦  Syntactic  and  semantic  descriptions 

♦  Parent  -  child  relationships 

•  Labeled  themes  and  concepts  are  the  architecture 
primitives 
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Topologies  in  Domains 
(information,  behavioral,  functional) 


•  Topology  in  architectures  relates  to  the 
connectedness  of  child  -  child  with  order  of 
precedence  and  importance 

•  Information  object  references  contain  topological 
reference  information  useful  in  describing  and 
identifying  the  syntactic  and  semantic  elements 

•  The  taxonomical  and  topological  elements  provide 
the  structure  and  precedence  of  concepts  and  their 
references  provide  the  content  for  specification 
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Objects. 
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S:V-  . 


AD V ANT ACt 
DEVELOPMEN 


Enterprise  Decomposition  by  Echelon 


1. De-confliction  of  meta-model 
components  to  remove  mixing  of 
concerns 

2. Encapsulation  represents  the 
statement  of  a  gestalt. 

3.1ncremental  instances  of 
encapsulation  represent  Rules  and 
States. 

^Decomposition  is  a  basic  principle 
of  Architecting  and  Engineering 


Decomposition  h/Echelon  hy Echelon:  as  two  separate  ‘synchronous’  taxonomies 
Functional  D^ompositjon  is  done  with  IDEFO,  and  the  Operational  Node 
Decomposition  is  dpue  synchronously  to  this 

Inpu^Output-ofthe  IDEF  0  Model  and  their  mapping  to  the  Node  Model  are  the 
ornjathfn  Flow  Model.  These  Inputs-Outputs  are  the  Information  Elements 
provide  topological  reference,  precedence  of  function  and  critical  exchange 
information  for  interoperability  concerns 

This  process  PROVIDES  THE  RIGOR  for  the  architecture  primitives. 
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View  Relationships:  Simplified  Calculus 


A  System  Function  done  at  a 
System  produces  an  Output 
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DISCOVERY  ENABLED 
ARCHITECTURE  SPECIFICATION 


•  Document  information  objects  describe  the  taxonomy  and 
topology  of  architecture  primitives  and  relationships 

•  Integration  is  accomplished  using  the  principles  and 
practices  of  a  tightly  coupled  discovery-based  architecting 
process 

♦  Indexing  and  Clustering  provide  navigation  to  the  authoritative  and 
original  sources  for  descriptions  of  the  information  objects 

♦  Clustering,  using  these  descriptions,  iteratively  refines  and  extracts 
more  relevant  information  objects 

This  enables  the  Synthesis  of  Form 

•  Discovery  described  Architectures  enables  the  development 
of  Rigorous,  Semantically  complete  Architecture 
Specifications,  i.e.,  engineerible  requirements 

This  enables  the  Analysis  of  Function 


©2008 

925  Bassett  Road,  Suite  A 
P.  O.  Box  45154,  Westlake,  OH  44145 


QUESTIONS 


Advantage  Development,  Inc. 
Michael  R.  Collins 
440-808-1250  Office 
216-570-8775  Cellular 
mcollins@advan-devel.com 
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Introduction 


Systems  Engineering  evolved  because  of  the  complexity  in  large 
scale  engineering  problems,  which  is  a  reality  of  today's  projects 

■  Transformation  to  Network  Centric  Operations  is  another  perturbation 
to  the  increase  in  complexity 

■  Program  Executive  Office  (PEO)  is  the  foundation  of  DoD  material 
development  that  produces  complex  systems  and  system  of  systems 

■  Additionally,  the  PEO  tends  to  be  a  complex  organization 

□  They  tend  to  be  large,  heterogeneous,  exercise  control  over 
strategic  objectives,  and  consist  of  portfolio  of  projects 

□  A  PEO  is  often  composed  of  several  Project  Managers  with 
their  own  complex  set  of  systems  engineering  challenges 

■  Intuitively,  we  understand  that  systems  engineering  at  the  Project 
Manager  (PM)  level  benefits  producing  complicated  systems 

■  What  form  should  systems  engineering  look  like  in  a  complex 
organization  such  as  a  PEO? 
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Complexity  —  effects  on  systems 


■  Systems  become  open 

■  Systems  behaviors  aren’t  reducible  to  the 
sum  of  their  parts 

■  Systems  parts  interact  nonlinearly 

■  More  difficult  to  completely  comprehend 
systems 

■  Is  a  fundamental  reason  for  failure  in  large 
scale  engineering  projects3 
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Complex  Systems 


■  Focus  is  on  the  overall  coherence  of  the  whole 
complex  system  -  without  direct,  immediate 
attention  to  the  details  while  typical  engineering 
tends  to  focus  on  the  functional  description 

■  Emphasis  is  on  how  decisions  are  made  and  not 
what  those  decision  should  be 

□  The  order  and  complexity  of  the  solution  rather  than  a  pre¬ 
specified  solution 

□  What  parts  of  the  whole  solution  should  addressed 

■  Relationship  and  interaction  of  the  population 
associated  with  the  complex  system  development  is 
key 
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Complex  Systems  (cont) 


■  Development  characteristics-  Shapes  the 

environment  and  not  the  actual  development 

□  Variety  of  autonomous  agents.  Ability  to  add  and  remove 
agents  without  halting  the  system 

□  Enable  autonomous  agents  interaction 

□  Resources  flow  throughout  the  development  without  any 
prescribed  means,  based  on  cooperation  and  competition 

■  Operational  characteristics 

□  Because  complex  systems  evolve  -  direct  interaction  is 
needed  between  development  and  operational 

□  Only  non  complex  systems  can  be  treated  in  a  way  of 
isolating  development  from  operation 

■  Enterprise  is  a  complex  system 
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Systems  Engineering 


Development  Domain 
Behaviors-  definable 


□ 


□ 


Outcome/Reward  -  predictable/static 
Acquisition  Domain 

□  Scope  -  linear/closed  boundary 

□  Budget/Financial  -systems  owned 
Human  Domain 

□  Human  Capital  -  skills  are 
understood  (classical) 

□  Organization-defined  &  structured 
Operational  Domain 

□  External  Systems-  single  interface 

□  Stakeholders-  single  user  class 


Development  Domain 


Complicated 


Complex 
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Acquisition  Domain 


System  of  Systems  Engineering 


■  Development  Domain 

□  Behaviors-  identifiable 

□  Outcome/Reward  - 
predictable/dynamic 

Acquisition  Domain 

□  Scope  -  linear/complicated 
boundaries 

□  Budget/Financial  -  systems  shared 

■  Human  Domain 

□  Human  Capital  -  skills  are  diverse 

□  Organization-  complicated  & 
relational 

■  Operational  Domain 

□  External  Systems-  multiply  systems  - 
similar  interfaces 

□  Stakeholders-  similar  users 


Development  Domain 


Human 


Domain 


Complicated 


Complex 
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Acquisition  Domain 


Enterprise  Systems  Engineering  (ESE) 


■  Development  Domain 

□  Behaviors-  self  organizing/open 

□  Outcome  -  adaptable/flexible 

■  Acquisition  Domain 

□  Scope  -nonlinear/open  boundary 

□  Budget/Financial  -  systems 
advocacy 

■  Human  Domain 

□  Human  Capital  -  skills  are  diverse 

□  Organization-  distributed  & 
cooperative 

■  Operational  Domain 

□  External  Systems-  multiple 
systems  -  multiple  interfaces 

□  Stakeholders-  multiple  users 


Development  Domain 


Human 


Domain 


Complicated 


Complex 
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Acquisition  Domain 


Systems  Engineering  in  Complex 
Organization  —  Use  Case  Example 

■  Program  Executive  Office  -  Simulation,  Training, 
and  Instrumentation  (PEO  STRI) 

□  Complex  Organization 

□  Complex  Systems  and  System  of  Systems 

■  Conceptual  application  of  “enterprise-level”  Systems 
Engineering  best  practices  to  support  the  PEO’s 
SoS  problem  space  of  integrating  the  Live  Virtual 
Constructive  (LVC)  domains. 

□  Utilized  SE  technical  management  processes  such  as 
technical  planning,  requirements  management  and 
interface  management 
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PEO  STRI  Organization 
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d. 


Policy  _  Systems 


PEO  STRI  Mission-Complexity  Space 


■  Provide  a  modular,  agile,  simulation, 
training,  testing,  and  instrumentation 
environment  to  enable  Warfighter 
success  for  any  threat 

■  We  must  capitalize  on  the  Army’s 
investment  through  integration  and 
interoperability  (12) 

■  Achieved  through  leveraging  and 
reuse  of  capabilities  across  the 
PEO’s  Enterprise  (i.e.:  Standards, 
Common  Products,  etc.) 

■  Provide  effective  and  efficient 
lifecycle  managements  of 
simulations  solutions  to  support  the 
Warfighter 
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A  Complex  System  —  LVC  Interoperability 


Live,  Virtual,  Constructive 
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LVC  12  —  ESE  Framework  Example 


LVC  Integration  &  Interoperability  (12) 
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Applying  ESE  —  Developmental  Environment 


Single  most  basic  activity  underpinning  engineering  enterprise 
systems 

■  Create  environment  of  continuous  innovation  to  address 
complexity 

■  PEO  STRI  established  a  group  called  the 
Integration/Interoperability  Advisory  Board  (I2AB)  to  provide 
governance  to  technical  and  PM  processes 

□  I2AB  provided  the  forum  for  team  organization  and  open 
communications  across  the  PEO  domains 

□  Comprised  of  technical  and  programmatic  leaders  from  each  of 
the  LA//C  domains 

■  I2AB  creates  coherence 

□  Requirements  Management 

□  Interface  Management 
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12  Advisory  Board  (I2AB)  Characteristics 

■  Responsible  to  provide  management  oversight,  direction, 
and  guidance  of  12  mission. 

■  Comprised  of  both  PEO  technical  and  program  senior 
leadership  and  reports  to  the  PEO  Board  of  Directors  (BOD). 

■  Provides  technical  and  program  recommendations  to  the 
DPEO/  BOD  to  facilitate  12  across  the  PEOs  program 
portfolio. 

■  Manage  the  PEO  portfolio  Dependency  Matrix. 

■  Establishes  12  standards,  guidelines,  and  processes  for  use 
and  compliance  in  coordination  with  PMs. 

■  Defines  12  policies  for  PEO  implementation. 

■  Educates  community  on  12. 
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Requirements  Management 


■  I2AB  understood  basic  complexity  principal  to  cope 
with  SoS  complexities  requires  increased  flexibility 

■  SoSE  requires  adaptation  to  changing  requirements 

■  Utilize  DODAF  to  develop  “enterprise”  architecture 
artifacts  to  support  interoperability  and  information 
exchange  requirements  for  LVC 

□  Methods  and  information 

□  Functions,  processes,  activities,  data  elements 

□  Standards 
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Interface  Management 

■  I2AB  understood  the  importance  of 
“standards-compliance”  as  an  asset  to 
support  interoperability 

■  I2AB  developed  and  enforced  the  use  of  the 
PEO’s  Common  Standards,  Products, 
Architectures  and  Repository  (CSPAR) 

■  Initiated  the  Live,  Virtual,  Constructive 
Integration  Cell  (LVCIC)  effort  to  begin 
integration  of  key  systems/interfaces  for  the 
LVC  Integrated  Training  Environment  (ITE). 
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ESE  LVC  Outcome  Challenges 

■  Data  Model  Strategy  that  supports  efficient  LVC  Training  and 
Testing  -  modeling  across  systems 

□  Fair  fight 

■  Consensus  on  what  is  “good  enough”  -  defining  the  “right” 
MOE/TPMs  that  apply  to  the  SoS 

□  Use  LVC  Interoperability  Model  as  “measuring  stick” 

□  Ensure  fidelity  and  density  of  data  and  signals  meets  needs  of 
both  test  and  training  communities 

□  Address  security  of  data  issues  across  all  communities 

■  Defining  clear  LVC  use  cases 

■  Resources  that  specifically  address  LVC  requirements 

■  Common  Test  /  Training  Solutions 

■  Scalability  of  LVC  products  -  Different  requirements  for  each 
domain 
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Summary 

■  Complexity  -  Impact  on  Systems 

□  Complex  organization  are  complex  systems 

□  Complex  System  are  open 

□  Complexity  makes  it  more  difficult  to  completely  comprehend  a  system 

Why  Enterprise  Systems  Engineering? 

□  Complex  systems  don’t  decompose  well  and  tend  to  be  nonlinear 

□  Complex  systems  behaviors  are  not  predictable 

□  Therefore,  classical  systems  engineering  approaches  need  modification 

■  Keystone  concepts  to  ESE  approach 

□  Configure  for  the  context  and  local  interaction  and  not  detailed  design 

□  Incorporate  processes  to  handle  unforeseen  changes  in  behavior 

□  Include  multiple  methods  for  achieving  the  same  end 

■  Potential  Benefits  to  PEOs 

□  Complex  systems  that  are  flexible  and  adaptable 

□  Ability  to  evolve  systems  through  introduction  of  new  technology  with  out 
disrupting  the  systems 

□  Ultimately,  reduces  risks  caused  by  unanticipated  effects  that  lead  to  failures 
of  systems 
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The  Situation 


•  Historically  for  warfighting  systems.... 

System  and  SoS  performance  =  f  {warfighter  performance} 

•  Future  for  warfighting  systems.... 

Dependence  Dependence  on  & 

Complexity  f  on  Netcentric  f  Criticality  of  Warfighter 

Environment  Performance 

•  Future  Warfighter’s  performance  =  f  {situational  awareness  (SA)} 

•  Future  Warfighter’s  SA  will  be  highly  dependent  on: 

-  Sensor  Input  -  Education  and  Training 

-  Information  from  Other  Humans  -  Combat  Experience 

-  Information  Systems  Output  -  Cognitive  Capabilities 

-  Others . 


The  Problem 


•  DOD  lacks  capability  to  measure  human  performance 

-  In  an  objective,  quantifiable  manner 

-  In  an  operational  environment  -  near  real  time 

-  With  statistical  quality 

•  Significant  shortcomings  in  measuring  a  warfighter’s  cognitive  SA. 

-  Much  progress  in  measuring  technical  SA 

•  Tracking  information  displayed  on  screens  or  available  in  a  network 

-  Limited  success  in  measuring  cognitive  SA 

•  In  a  laboratory  environment 


•  Limited  technical  means  for  collecting  objective  data  in  support  of 
assessing  cognitive  SA  in  an  operational  environment 


As  the  complexity  of  systems  and  level  of  information  flow 
increases,  this  assessment  deficiency  grows  proportionately  larger 


The  Problem 

(Continued) 


Complex 


Acceptable  ability  to 
test  in  an  operational 
environment 


Weapons 

Sensors 

Platforms 

Munitions 

Terrain 

Weather 


•  Decision  Making 

•  Intelligence 

•  Communication 

•  Inspiration 


\ 


Inadequate  ability  to 
test  in  an  operational 
environment 


/ 


Morale 

Training 

Confidence 

Fatigue 

Fear 

Risk  Aversion 


Limited  ability  to  test  all  aspects  of  a 
Warfighter’s  combat  environment 
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The  Program 


Joint  Warfighter  Test  and  Training  Capability  (JWTTC) 

•  A  major  US  Army  major  instrumentation  program 

•  Focused  on  measuring 

-  Cognitive  human  performance 

-  Cognitive  SA 

-  Physiological  status 

-  In  an  operational  environment 

•  Will  address  test  and  evaluation  (T&E)  shortfalls  in  terms  of 

-  Instrumentation 

-  Measurement  and  analysis  of  Warfighter  performance 

-  Impact  of  physiological  and  neurological  stress 

-  The  collection  and  analysis  of  Warfighter  performance  data  in  terms  of 

•  SA  of  an  individual 

•  Shared  SA  (SSA)  of  teams,  crews,  or  combined  teams  and  crews 

•  The  total  system  performance  of  a  single  manned  system  or  a  combination 

of  Warfighters,  manned  systems,  and  unmanned  systems.  5 


Systems  Engineering  (SE) 

Need  for  JWTTC 


•  DOD  5000.2  requires  systems  engineering  in  a  program’s 
acquisition  life  cycle 

•  The  SE  describes  the  overall  technical  approach  to  development  of 
an  effective  JWTTC  product  that  is  sustainable  at  an  affordable  cost 

•  Identifies  how  the  program  is  structured  and  conducted  to  effectively 
achieve  program  goals  and  objectives 

•  It  an  instance  of  the  technical  baseline  defining  the  architecture  and 
design  components 

-  Decomposes  the  capabilities  into  logical  and  physical  components 

-  Includes  technical  performance  measures 

•  Provides  the  road  map  for  acquiring  and  integrating  technologies  to 
address  the  JWTTC  capabilities 

-  Includes  a  comprehensive  program  schedule  outlining  component 
acquisition  activities,  integration,  test,  and  delivery 

•  A  tool  in  managing  the  technical  development  of  JWTTC  System 
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Engineering  Approach  for 

JWTTC 


•  Consideration  in  developing  the  JWTTC 
program 

-  Warfighter  is  a  system  in  JWTTC 

-  JWTTC  is  a  system-of-systems 

•  Use  proven  SE  approaches  to  evaluate 
the  systems 


Warfighter  as  a  Node  in 
an  SoS  Environment 
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Warfighers 
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The  JWTTC  SoS 
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Use  Proven  SE 
Approaches 


•  Support  the  development  of  JWTTC 

-  Use  a  systems  approach  to  develop  the  program 

-  Conduct  a  systems  engineering  analysis  effort 

•  To  identify  system  requirements 

-  Through  Use  Cases 

-  Through  decomposition  of  evaluation  metrics 

•  To  develop  a  system  architecture 

-  Develop  a  Systems  Engineering  Plan  (SEP) 

-  Implementing  the  SE  process 

-  Integrate  SE  effort  with  the  overall  program 
management  control  efforts 
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Operational 

Concept 


JWTTC 

Concept 


Concept 

Refinement 


The  SE  Effort 

Systems  Approach  for  Developing  JWTTC 


System  Requirements 


Operational 
Use  Case  |— 
Scenarios 


System 

Capabilities 

(Functional) 


System 

Capabilities 

(Physical) 


System 

Requirements 


System 

Requirements 

Review 


System 

Design 

Architecture 
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1 

Technology  Demonstration 


Design 

Considerations 


System 
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System  Architecture 


11 


The  SE  Effort 

Identifying  System  Requirements  (Approach  #1 ) 


•  Develop  Use  Cases 

-  Narrative  descriptions  of  a 
sequence  of  activities  a  T&E 
effort  would  undertake 

-  Use  cases  do  not  identify 
capability  needs,  but  rather 
imply  them  in  the  story  it  tells 

-  An  analyst  then  identifies 
capability  needs 

•  Derive  requirements  from 
the  capability  needs 


•  Top  Level 

-  Actors 

• 

IT  Systems 

• 

Warfigther 

• 

Test  Control 

• 

Test  Environment 

-  Cases 

• 

Pre  test 

• 

Test 

• 

Post  test  data 
collection  (e.g.,  AAR) 

• 

Data  Transfer 

• 

Post  Test  Analysis 

• 

Failure  Warning 
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The  SE  Effort 

Identifying  System  Requirements  (Approach  #2) 


tCellence  in  Test 


^rdeen  Test  Ce0 


Decompose  evaluation  metrics  (e.g.,  measures  of 
effectiveness) 

Measure  of  Effectiveness 

l 

What  Needs  to  be  Assessed 
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-►  Reqt  1 

-►  Reqt  1 

-►  Reqt  1 

-►  Reqt  1 

-►  Reqt  1 

-►  Reqt  1 

-►  Reqt  1 

-►  Reqt  1 

-►  Reqt  2 

-*  Reqt  2 

-►  Reqt  2 

-►  Reqt  2 

-►  Reqt  2 

-►  Reqt  2 

-►  Reqt  2 

-►  Reqt  2 

-►  Reqt  n 

-►  Reqt  n 

-►  Reqt  n 

-►  Reqt  n 

-►  Reqt  n 

-►  Reqt  n 

-►  Reqt  n 

-►  Reqt  n 

The  SE  Effort 

Defining  the  System  Architecture 


•  Once  requirements  are  identified,  design 
an  architecture  that  satisfies  the 
requirements 

•  Conduct  experiments  of  the  architecture 
design  using  functioning  systems, 
prototypes,  and  surrogates 

•  Adjust  the  architecture  as  needed 

•  Identify  areas  of  risk  and  potential 
mitigation  efforts 
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The  SE  Effort 

Defining  the  Functional  Architecture 
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The  SE  Effort 

System  Engineering  Plan  (SEP) 


•  The  JWTTC  SE  methodology  is  tailored  from  the 
ISO/ECI  15228  four  systems  engineering 
process  groups  (Technical,  Project,  Enterprise, 
Agreement) 

•  The  tailored  JWTTC  SE  methodology  includes 

-  Technical  processes 

•  Requirements  development,  logical  analysis,  design  solution, 
impiementation,  and  integration 

-  Parts  of  the  project  processes 

•  Decision  making 

•  Risk,  configuration,  and  information  management 

-  Enterprise  environment  management  process  groups 


16 


The  SE  Effort 

Implementing  SE  Processes 


•  As  described  in  the  SEP,  the  plan  is  to 
implement  JWTTC  SE  processes  using  the  Vee 
systems  engineering  method 
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The  SE  Effort 

Integrate  SE  Effort  with  Overall  Program 
Management  Control  Efforts 
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In  Closing.... 


•  Much  of  the  JWTTC  Systems  Engineering 
effort  is  being  refined 

•  The  approach  so  far  has  been  beneficial  in 
enhancing  the  JWTTC  program 

•  The  effort  should  prove  to  be  an  effective 
method  for  reducing  JWTTC  program  life 
cycle  risks  due  to 

-  Complexity  of  the  technology 

-  Unforeseen  changes 
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NDIA  CMMI®  Working  Group 


IMiMMMtiiai 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Charter 

•  Collect  and  provide  a  broad-based,  representative  viewpoint  on  issues 
relating  to  CMMI-based  process  improvement  within  NDIA  member 
companies 

•  Advise  NDIA  SE  Division  and  CMMI  Steering  Group  on  CMMI  Product  Suite 
content,  issues,  and  strategies  for  implementation,  appraisal,  and  training 
with  recommendations  to  optimize  the  leverage  of  CMMI  investments  in 
government  and  industry 

Membership 

•  Representatives  from  industry,  government,  academia,  and  SEI 
(see  membership  list) 

Tasking 

•  Respond  to  requests  for  input  from  CMMI  Steering  Group 
(product  reviews,  position  papers,  recommendations,  feedback) 

•  Provide  bi-directional  communications  and  feedback  from  CMMI  community 

CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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NDIA  CMMI®  Working  Group 
Interfaces  and  Work  Flows 
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CMMI  Product  Team 
Transition  Partners 
Partner  Advisory  Board 
Config.  Control  Board  (CCB) 
Expert  Groups 

h  maturity,  appraisals,  etc.) 
•Advisbry  Board  (SLABOK,  etc.) 
•Other  (ass^pplicable) 

SEI 

Working 
Groups 


CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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CMMI WG  Membership 


mmsmsMsssMmmMm 

Ninam 

arn'Mimi  mum  ivni  vi  u\  *  n/niKm  iv:v 


Name 

Organization 

Jim  Armstrong 

Stevens  Institute 

Karen  Bausman 

USAF  AFIT 

Dan  Blazer 

SAIC 

Geoff  Draper  (lead) 

Harris  Corporation,  Govt  Communications  Systems  Division 

Jeff  Dutton 

Jacobs  Technology  Inc. 

Ray  Kile 

Lockheed  Martin,  Systems  and  SW  Resource  Center  (SSRC) 

Dawn  Littrell 

L-3  Communications 

Wendell  Mullison 

General  Dynamics,  Land  Systems 

Randy  Walters 

Northrop  Grumman  Mission  Systems,  C2  Systems  Division 

Jon  Gross 

Software  Engineering  Institute  (SEI) 

Mike  Phillips 

Software  Engineering  Institute  (SEI) 

Karen  Richter 

Institute  for  Defense  Analyses  (IDA) 

NDIA  Systems  Engineering  Conference 
October  2008 


4 


CMMI  WG  Organization 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Subteam 

Summary  Objectives 

Membership 

High  Maturity 

(HiMat) 

Subteam 

'Respond  to  SG  priority  direction  on  HiMat  issues 
'Provide  industry  input  on  CMMI  L4-L5  model 
issues  and  process  improvement  benefits 

Randy  Walters  (lead  -  NG) 
Wendell  Mullison  (GD) 

Jim  Armstrong  (Stevens) 

Ray  Kile  (LM) 

Dan  Blazer  (SAIC) 

Dawn  Littrell  (L-3  Com) 

(Karen  Richter:  OSD  liaison) 

CMMI  Survey 
Subteam 

'Collect  broad-based  industry  feedback  on  CMMI 
via  conference  sessions 

Geoff  Draper  (lead  -  Harris) 

Jeff  Dutton  (Jacobs) 

Karen  Bausman  (USAF) 

CMMI 

Performance 

Subteam 

'Quantify  CMMI  performance  improvements 
'Linkage  between  CMMI  MLs  and  program 
performance 

Jeff  Dutton  (lead  -  Jacobs) 

Karen  Bausman  (USAF) 

Wendell  Mullison  (NG) 

Randy  Walters  (NG) 

Task  descriptions  validated  with  CMMI  Steering  Group 
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CMMI®  Interactive! 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Did  you  ever  want  a  voice  on  what  works,  and  what  doesn’t,  with 
the  implementation  of  CMMI  in  industry? 

Objective: 

•  Collect  and  provide  real-time,  interactive  feedback  on  how  well  your 
organization's  implementation  of  CMMI  supports  the  business 
objectives  within  your  organization 

Approach: 

•  Live  anonymous  electronic  voting  and  results  analysis 

•  Results  will  be  provided  to  CMMI  Steering  Group  and  SEI  to  help 
establish  future  directions  for  the  CMMI  Product  Suite 

•  No  areas  are  off  limits! 

-  Model,  appraisals,  training,  business  impact,  .... 

•  Open  discussion  for  additional  feedback  (as  time  permits) 

Appreciation  to  Harris  Corporation  for  use  of  interactive  voting  devices. 
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What  type  of  organization  are  you 
representing? 


1.  Defense  Industry 

2.  Commercial  Industry  (U.S.) 

3.  Commercial  industry  (Non-U. S.) 

4.  U.S.  Government 

5.  FFRDC 

6.  Academia 

7.  Other 
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Does  your  organization  have  a  CMMI 
maturity  level  rating? 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  ML  1 

2.  ML  2 

3.  ML  3 

4.  ML  4 

5.  ML  5 

6.  No  rating 


46% 
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How  large  is  your  organization  (staff  size)? 

(for  the  organizational  unit  with  the  CM  Ml  maturity 
level  rating  indicated  previously) 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  <  25  people 

2.  25-100  people 

3.  100-500  people 

4.  500-1000  people 

5.  1000-5000  people 

6.  5000-10,000  people 

7.  >  10,000  people 


31%  31% 
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Does  your  organization  have  defined  goals  for 
achieving  a  CMMI  maturity  level  rating? 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  ML  1 

2.  ML  2 

3.  ML  3 

4.  ML  4 

5.  ML  5 

6.  No  specific  level  targeted 


50% 
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How  much  confidence  do  you  have  in  CMMI 
maturity  level  ratings  as  benchmarks? 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  Very  high  confidence 

2.  High  confidence 

3.  Moderate  confidence 

4.  Little  confidence 

5.  No  confidence 


50% 


36% 


0% 


7%  7% 
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How  representative  is  your  maturity  level  rating 
of  how  projects  really  execute  in  your 
organization? 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1 .  Very  representative  (all  projects) 

2.  Mostly  representative  (most  projects) 

3.  Somewhat  representative  (some  projects) 

4.  Marginally  representative  (few  projects) 

5.  Not  representative  (no  projects) 


42% 
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How  much  business  value  has  your  organization 
obtained  through  deployment  of  CMMI? 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  Very  high  value 

2.  High  value 

3.  Moderate  value 

4.  Marginal  value 

5.  Low  value 

6.  None 


38%  38% 
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What  grade  would  you  give  the  CMMI 
Product  Suite  overall  in  meeting  the  needs 
of  your  business? _ 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  A+ 

2.  A 

3.  B 

4.  C 

5.  D 

6.  F 

7.  Incomplete 


42% 
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What  relationship  has  improvement  in 
CMMI  maturity  levels  had  on  performance 
of  projects  in  your  organization? 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  Very  high  positive  impact 

2.  High  positive  impact 

3.  Moderate  positive  impact 

4.  Little  to  no  impact 

5.  Moderate  negative  impact 

6.  High  negative  impact 

7.  Very  high  negative  impact 


36% 
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What  is  the  primary  reason  your 
organization  uses  CMMI? 


1.  Maturity  level  needed  to  bid  on 
contracts 

2.  Competitive  advantage  from 
maturity  level  ratings 

3.  Improvement  of  business 
processes 

4.  Corporate  standardization 
initiative 

5.  Leverage  best  practices  proven 
successful  in  industry 

6.  Other 
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0% 


18% 


What  are  the  top  benefits  your  organization  has 
realized  from  implementation  of  the  CMMI? 
(Pick  up  to  3  choices  in  priority  order) 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1 .  Performance  -  Consistently  enhanced  project 
performance 

Marketing  -  Better  marketability/win  rate 
Predictability  -  Enhanced  ability  to 
accurately  predict  project  performance 
Program  Startup  -  Enhanced  ability  to  “start 
up”  a  new  project/program  in  a  repeatable 
and  predictable  manner 
Responsiveness  -  Enhanced  ability  to  react 
to  customer  risks  with  processes  tailored  to 
the  customer’s  needs 

Cycle  Time  -  Decreased  timelines  for  product 
development  life  cycles 

Customer  Satisfaction  -  More  satisfied 


50%  50% 


2. 

3. 

4. 


5. 


6. 


7. 


8. 


10. 


customers  and  more  repeat  business 
Quantitative  Management  -  Enhanced  ability 
to  “tell  our  story”  in  a  defined,  quantitative 
manner 

Employee  Morale  -  satisfied  employees, 
reduced  turnover 

Human  Capital  -  More  highly  skilled  and 
knowledgeable  employees 
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What  are  the  top  issues  related  to  the 
effectiveness  of  CMMI? 

(Pick  up  to  3  choices  in  priority  order) 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1 .  Gaming  -  maturity  levels  undeserved 

2.  Implementation  Cost  -  Too  costly  to 
implement  CMMI 

3.  Appraisal  Cost  -  Too  costly  to  do 
appraisals 

4.  Inaccuracy  -  Appraisal  results  are  not 
accurate 

5.  Not  Useful  -  CMMI  content  is  not  useful 
for  my  type  of  business 

6.  Low  Value  -  The  overall  return  does  not 
justify  the  investment  (low  ROI) 

7.  Complexity  -  Model  is  too  large  (too 
many  process  areas  and  practices) 

8.  Wrong  Emphasis  -  Too  much  emphasis 
on  compliance,  not  enough  on 
improvement 

9.  Consistency  -  Inconsistent  model 
interpretations 

10.  No  issues  -  CMMI  works  fine  in  my 
organization 


54% 


31% 


46% 


46% 


8%  8% 
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What  should  be  the  top  priorities  for  improving 
the  CMMI  Product  Suite? 

(Pick  up  to  3  choices  in  priority  order) 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  Lean  the  model  (make  it  smaller) 

2.  Lean  SCAMPI  (streamline  the  method 
and  evidence  rules) 

3.  Provide  better  training 

4.  Add  more  disciplines  (new  model  PAs 
or  constellations) 

5.  Make  appraisals  more  efficient 

6.  Enforce  appraisal  quality  (less  gaming) 

7.  Clarify  high  maturity  practices  (CMMI 
ML4-ML5  PAs) 

8.  Provide  better  linkage  between  process 
capability  and  project  performance 

9.  Provide  more  SEI  support 

(e.g.,  resources,  examples,  assets) 

10.  Nothing;  it’s  fine  the  way  it  is 


54% 


38% 


46% 


38% 


23% 
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Are  you  representing  an  organization  that 
actually  develops  products? 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 
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CMMI  -  Open  Discussion/Feedback 


UMft 

mmsmmsmmmima 

Ninam 

STRENGTH  THROUGH INDUSTRY  &  TECHNOLOGY 


What  Works? 

What  Doesn’t? 

•CMMI  does  not  include  business 
results 

•ISO/Baldrige  is  more  objective  - 
appraisals  must  be  completely 
objective  and  independent  (not 
people  appraising  their  own  work) 
•CMMI-SVC:  draft  appears  more 
ITIL/SW/IT  oriented;  does  not  well 
support  government  services 
organizations,  SETA 
•Model  should  focus  more  on 
measurable  results;  must  be 
important  to  the  organization,  show 
positive  trends 
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Thank  you  for  your  participation! 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Watch  for  more  communications  feedback. 

Want  to  learn  more  or  get  involved? 

Contact  your  CMMI  Working  Group  representative,  or: 

Geoff  Draper 
Harris  Corporation 
gdraper@harris.com 


Please  return  the  interactive  voting  devices! 
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Systems  Engineering 
Capability  Development 
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Systems  Engineering 
Capability  Development 


Overview 

•  The  application  of  disciplined  Systems  Engineering  has  been 
proven  to  significantly  improve  program  performance 
especially  on  complex  systems. 

•  This  fact  is  particularly  important  for  Department  of  Defense 
programs  which  are  often  large  scale  and  complex. 

•  The  quickest  way  to  realize  systems  engineering  benefits  is 
to  prioritize  work  efforts  based  on  the  highest  return  on 
investment. 

•  One  key  step  to  success  is  for  an  organization  to  benchmark 
their  own  Systems  Engineering  capability,  identify  gaps,  and 
plan  to  improve. 

•  This  session  will  discuss  an  analytical  approach  for  rapidly 
maturing  Systems  Engineering  capability  within  institutions 
as  applied  across  multiple  programs  and  lifecycle  phases. 
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RDECOM 


Increased  Complexity  Demands 
Increased  SE  Capability 


TF^r^UtEi 


Performance  vs.  PC  and  Overall  SEC 


Complexity  of  Current  and  Future  Systems 

Traditional  SE  Approaches  are  not  sufficient  to  tackle 
increasingly  large-scale  complex  systems 

The  SE  community  is  paying  increasing  attention  to  issues  of 
Systems  of  Systems,  complex  systems,  and  enterprise  systems 

Increased  system  complexity  warrants  increased  systems 
engineering  capabilities.  Considerations  include: 


Source:  Software  Engineering  Institute  and  NDIA  -  Elm, 
Joseph  P.,  et  al.  A  Survey  of  Systems  Engineering 
Effectiveness — Initial  Results ,  November  2007 


Agile  Constructs  and  Lean  Processes  for  rapid  execution 
Integrating  technologies  across  multiple  Families  of  Systems 
Increased  demands  requiring  optimal  trades/balancing 
System  of  Systems  Analysis,  Interoperability,  constrained  integration 


Mission 

Dynamics 


Ground  Domain  Complexity 

■  TARDEC  SE  Applications 

■  Science  and  Technology  Programs 

■  Mine  Resistant  Ambush  Protected  (MRAP) 

■  -  Required  speed  of  execution  &  trades  for  survivability 

■  Condition  Based  Maintenance 

■  Technology  Integration  across  multiple  families  of  systems 

■  Joint  Lightweight  Tactical  Vehicle 

■  Large  new  program  seeking  to  balance  Payload  -  Protection  - 
Performance 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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TARDEC  MISSION  AND  VISION 


MISSION:  TARDEC  develops,  integrates,  and 
sustains  the  right  technology  solutions  for  all 
manned  and  unmanned  DOD  Ground 
Systems  and  Combat  Support  Systems  to 
improve  Current  Force  effectiveness  and 
provide  superior  capabilities  for  the  Future 
Force 


VISION:  The  recognized  DOD  lead  for  Ground 
Systems  &  Combat  Support  Systems 
Technology  Integration  and  Systems  of 
Systems  Engineering  across  the  Life  Cycle 
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Heavy  Tactical 
Vehicle 


Light  Tactical  Vehicle 


Medium  Tactical  Vehicle 


Countermine  Equipment 


Combat  Vehicles 


Water  Generation 
and  Purification 


Fuel  and  Water  Storage  & 
Distribution  Quality 
Surveillance  Equipment 


Watercraft 


Logistics  Equipment 


TARDEC  is  responsible  for  research,  development  and  engineering  support  to  more  than  2,800  Army 
systems  and  many  of  the  Army’s  and  DoD’s  top  joint  warfighter  development  programs. 


RDECOM 


SE  Revitalization 


TRRDEI 


■  The  Department  of  Defense  (DOD)  and  the  Department  of 
the  Army  (DA)  have  promoted  the  revitalization  of  SE  and 
have  issued  SE  Policies  aimed  at  the  acquisition 
community. 

■  Under  Secretary  of  Defense  Acquisition,  Technology  and  Logistics  Policy  for 
Systems  Engineering  (SE)  in  Department  of  Defense  (DOD),  20  February 
2004,  Addendum  22  October  2004. 

■  Department  of  the  Army,  Office  of  the  Assistant  Secretary  of  the  Army 
Acquisition,  Logistics  and  Technology  (ASA(ALT))  Army  Systems 
Engineering  (SE)  Policy,  13  June  2005. 


■  RDECOM  &  TARDEC  has  also  issued  a  SE  Policy 
applying  SE  discipline  to  Science  &  Technology 
programs. 

■  U.S.  Army  Research,  Development  and  Engineering  Command  (RDECOM) 
Systems  Engineering  (SE)  Policy,  24  April  2007 

■  TARDEC  Systems  Engineering  (SE)  Policy,  27  September  2007 


All  programs  shall  apply  a  robust  SE  approach  that 
balances  system  performance  with  total  ownership  costs 
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RDFnon/T^)  Typical  Challenges 


■Organizational 

■  Isolated  pockets  of  SE  practice 
■Competing  stove  piped  processes 

■Lack  of  integration  with  business  and  management  practices 
■Organizational  Alignment  to  enable  SE 

■Application  of  SE 

■Across  the  lifecycle  (concept  through  disposal) 

■Science  and  Technology  Programs 
■Limited  Budget 

■Synchronization  Across  Programs 

■Misconceptions 

■Assign  an  SE  to  a  Project  &  Systems  Engineering  Will  Get  done! 
■Train  and  Certify  the  Workforce  in  SE  and  SE  Will  Get  done! 
■Take  a  Ride  on  the  SE  “V”  (diagram)  and  SE  Will  Get  done! 

■SE  Definition 

■  Everything  is  SE! 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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SE  Framework 


TH/TDEI 


SE  Policies 
Industry  Standards 
Communities  of  Practice 


Voice  of  the  Customer 
Mission  Requirements 
Voice  of  the  Business 


SE  Centers  of  Excellence 
Products,  Services,  &  Innovations 


Established  an  SE  Framework  and  an  integrated  organizational 

structure  to  enable  SE! 
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**oecom^  Strategy  to  Building  SE  Capability 


■Define  and  Document  the  Requirements 

■Conduct  QFD  Sessions  to  Solicit  the  VOC 

■  Benchmark  Other  SE  Organizations/Efforts 

■  Leverage  DOD  /  Industry  /  Academia  Studies 

"Baseline  Capabilities 

■Establish  a  Baseline  of  TARDEC’s  Systems  Engineering 
Capabilities  and  Performance 

■  Identify  Areas  for  Improvement  and  Make  the  Business  Case 
for  Change  Based  on  Risks  and  Opportunities 

"Capability  Development  Plan 

■Build  a  focused  and  prioritized  work  plan  to  address  gaps 
■Leverage  Strengths  and  Best  Practices  from  Industry 
■Institutionalize  Systems  Engineering 
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Systems  Engineering  Capability 
&  Program  Performance 


7"nr?o£=cr 


Project  Performance  vs.  Systems  Engineering  Capability 


Projects  with 
Lower  £E 
Capability 


Pro j  eels  wi  t  h  Proj  ects  wi  t  h 

Moderate  st  Higher  £E 

Capability  Capability 


■  Higher  Project 
Performance 

□  Moderate  Project 
Peformance 

■  Lower  Project 
Performance 


Performance  vs.  PC  and  Overall  SEC 


Lower 

Capability 


Low 

Challenge 


Statistical  relationship  with  Project  Performance 
is  quite  strong  when  both  SE  Capability  and 
Project  Challenge  are  considered  together 


Study  demonstrated  that  projects  with 
better  Systems  Engineering  Capabilities 
delivered  better  Project  Performance. 


Supplier’s  Systems  Engineering 

Capability' 

Relationship  to 

Project  Performance 

Relationship 

(Gamma'1) 

Project  Planning 

Weak  positive  relationship 

+0.13 

Project  Mon  tor  ng  and  Contra 

Weak  negative  relationship 

-0.13 

Risk  Management 

Moderately  strong  positive  relation¬ 
ship 

+0.28 

Requirements.  Development  and  Management 

Moderately  strong  positive  relation¬ 
ship 

+0.33 

T rads  Stud  es 

Moderately  strong  positive  relation¬ 
ship 

+0.37 

Product  Architecture 

Moderately  strong  to  strong  positive 
relationsh  p 

+0.40 

Technical  Solution 

Moderately  strong  positive  relation¬ 
ship 

+0.36 

Product  Integral  on 

Weak  positive  relationship 

+0.21 

Verification 

Moderately  strong  positive  relation¬ 
ship 

+0.25 

Validation 

Moderately  strong  positive  relation¬ 
ship 

+0.28 

Configuration  Management 

Weak  positive  relationship 

+0.13 

IPT-Re  ated  Capabi  ity 

Moderately  strong  positive  relation¬ 
ship 

+0.34 

Source:  Software  Engineering  Institute  and  NDIA  -  Elm,  Joseph  P.,  et  al.  A 
Survey  of  Systems  Engineering  Effectiveness — Initial  Results ,  November  2007 
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Booz  Allen’s  Integrated  Acquisition  Capability™(IAC) 


Integrated  Acquisition  Capability™ 


T 


T 


Leadership 

j  Acquisition  j 

|  Program 

|  Technical 

Capability 

|  Capability  f 

Capability 

i  Capability^ 

1 .  Concept  Development 

2.  Acquisition  Strategy 

3.  Acquisition  Process 

4.  Major  Restructuring 

5.  Acceptance  &  Transfer 

6.  Follow-on  Business 
Development 

1  Program  Planning 

2.  Performance  Mgmt 

3.  Supplier  Management 

4  Logistics  Management 

5.  Schedule  Management 

6.  Financial  Management 

7.  Risk  Management 

1 .  Stakeholder  Requirements 
Definition 

2.  Requirements  Analysis 

3.  Architectural  Design 

4.  Implementation 

5.  Integration 

6.  Verification  (induding  T&E) 

7.  Transition 

8.  Validation 

7.  Life  Cycle  Approach 

8.  External  Stakeholder 
Management 

9.  Cost  Estimating 

10  Legal  &  Regulatory 
Compliance 

8.  Program  Organization 

9.  Contract  Management 

10.  Mission  &  Info  Assurance 

11.  Program  Review  Process 

12.  Configuration  &  Data  Mgmt 

13.  Infrastructure  Management 

9.  Technical  Performance 
Management 

10.  Modeling  &  Simulation 

11.  Technology  Mgmt 

12.  Development  Infrastructure 

1  Program  Strategy 

2.  Program  Authorities 

3.  Program  Operating  Model 

4.  Institutional  Knowledge 
Capture  and  Dissemination 


5.  People  Development  and 
Deployment 

6.  Process  Management 

7.  Tools  &  Infrastructure 
Support 


Integrated  Acquisition  Capability™  is  a  proprietary  methodology  and  trademark  of  Booz  Allen  Hamilton,  Inc. 


■  Depicts  the  complete  set  of  capabilities  required  to  successfully  execute  a  program 

■  Derived  from  multiple  industry  and  government  standards  as  well  as  extensive  experience 

■  Provides  a  common  framework  for  assessing  and  building  capabilities  across  industries 

■  The  IAC  is  a  proprietary  methodology  easily  tailored  to  each  unique  client  environment 


Systems  Engineering  Capability  Development 

Booz 

Allen 

Hamilton 
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Building  a  Systems  Engineering  Capability  Development  Plan 


INPUTS 

Q) 

•  Integrated  <§ 

Acquisition  :F 

Capability™  (IAC) 
Framework 


•  Direction  from 
Organization  on 
Focus  Areas / 
Scope 

•  Previous 
Assessments 

•  Organizational 
Chart  &  Contact 
List 


•  Program 
Documentation 
(e.g.  SE  artifacts, 
processes,  etc.) 


►  Preparation  Phase 

-  Setup  Assessment  Tool 

-  Tailor  Interview  Questions 

-  Tailor  Document  Review  List 

-  Schedule  Interviews  &  Collect 
Documents 


OUTPUTS 

•  Capability  Scorecard 

•  Identified  Key  Root 
Causes  for  Gaps 


►  Data  Collection  Phase 

=  -  Conduct  Interviews 

$  -  Perform  Document  Reviews 

o-  -  Explore  Cross  References 

-  Record  Findings 


•  Recommendations  for 
Improvement/ 
Prioritized  Improvement 
Plan 


•  Capability  Development 
Plan  Final  Brief  and 
Report 


►  Analysis  Phase 

-  Synthesize  Data 

-  Identify  Strengths  &  Weaknesses 

-  Score  Capabilities 

-  Document  Findings 


Process  to  build  an  SE  Capability  Development  Plan 


Systems  Engineering  Capability  Development 

Booz 

Allen 

Hamilton 
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Tailoring  of  the  IAC  Framework  &  Defining  Scope 


Integrated  Acquisition  Capability™ 


r 


T 


T 


Leadership 

Capability 


nuv^ui^iull  1 1 


1 


Capability 


Program  Strategy 
Program  Authorities 
Program  Operating  Model 
Institutional  Knowledge 
Capture  and  Dissemination 


i .  icepi  ueveii  p  Tient 

O  Arm  licitinn  Qfrta  g  gy 

Acquisition  Process 
Major  Restructuri  lg 
Acceptance  &  Trs  nsfer 
Follow-on  Busin  e  ss 
Development 


Program 

Capability 


Technical 

Capability 


Pla/]AiC 


ilder  Requirements 


5U| 

Lex 
Scliedule 
Fin  ancial 
Risk  Man 


Capability 

Definition 

Architectural 

Design 

Synthesize  a  system  solution 
that  satisfies  the  requirements. 

i  ransraon 


Capability:  Architectural  Design 


DoD  Systems  Engineering  Processes 

Technical  Management  Process 

Technical  Processes 

Decision 

Analysis 

Risk 

Management 

Requirements 

Development 

Integration 

Technical 

Planning 

Configuration 

Management 

Logical  Analysis 

Verification 

Technical 

Assessment 

Technical  Data 
Management 

Design  Solution 

Validation 

Requirements 

Management 

Interface 

Management 

Implementation 

Transition 

Source:  Defense  Acquisiti6fi%\^er^iy^ 


ology  and  trademark  ofBooz  Allen  Hamilton,  Inc. 


Program  Organization 
Contrac 
0.  Mission 

1 .  Program  Kevis^  Process 

■\nfini  irati  _! 


Technology  & 
Infrastructure 


A  complete  capability  requires 


)table  a  standardl  Process 


zes  from  soctotd 
yii  lybllliuiuljy  industries. 


he  right|People 
enabled  by  the 


progrargsrolberj  Technology  Infrastructure 


in 


ance  with  a  definedl  Governance 
mechanism. 


Systems  Engineering  Capability  Development 


Booz  I  Allen  I  Hamilton 


NDIA  1 1th  Annual  Systems  Engineering  Conference 
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Assessing  SE  Capability 
Preparation  Phase 


TR/TDE 


a>  o 

C 

S  o 
£  ^ 


•  All  Levels 
Non-Attributional 
360°  Perspective 


TFinnisi 


Interviewed 
Not  Available 
Briefed 


Interview  List 


Tailored  Questions 


I 


Schedule 

Interviews 


a>  ^ 


S  iS 
^  £ 


Setup  Assessment  Tool 


A 


OS~  O  Sr 


uTs» 


SEP,  TEMP  P-Spec,  IMP, 
IMS  etc. 


Tailor  Interview  Questions 
&  Document  Review  List 


Schedule  Interviews  & 
Collect  Documents 


ATO 

Manager’s 

Guidebook 


Tailored  Doc  List 


1 


^^1 


Capability  Artifacts 
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Assessing  SE  Capability 
Data  Collection  Phase 


Require 

ments 

Analysis 


Are  all  types  of  requirements 
(functional,  performance,  design 
constraints,  regulatory,  etc) 
captured  in  a  consistent  manner? 


-  r  1  ki-  iti. 


flli.U^Fe  «ii  lyn^-  i' 
i, .  dhM.Fi  * 


PJi>,  «ur  lv#ni  M$  Won)  rJwturFiunl  t,ir  nrggMwjf 

■Mjuiw-Enli  and  i  ftir  new  u*«r  r*. qL*imi?<ii'  Thf 


Notional  Example:  T earn  uses  a  text 
document  for  regulatory  requirements  and  a 
spreadsheet  for  new  user  requirements. 
Requirements  tools  are  available  but  not 
used,  RTM  used  by  test  did  not  include  most 
recent  changes. 


Record  Findings 


Al 

j/ 


Notional  Example:  02.01  "The  established 
reqmnts.  database  shall  be  the  authoritative 
requirements  management  tool  for  capture 
and  recording  of  new  requirements  to 
provide  full  traceability... .changes  must  be 
approved..” 


1 


* — 

T" 

— 50= — 

aYsl bins  IzhgiheeNhg  I  J\ - 

'The  established  red  As  database  shall  be 

2.01 

the  autoritative  regrJ/ements  management 
tool  for  capture,  recording  and  to  provide 

P  Spec 

SEP  " 

Systems 

5 

full  traceability." 

TEMP 

Test  &  E 

n 

6 

TEMP 

Test  &  Evaluation  Management  Plan 

ICD 

Initial  Ca 

I® 

8 

ICD 

Initial  Capabilities  Document 

AoA 

Analysis 

+ 

10 

TP 

Test  Report  (Results) 

TP 

Test  Ref 

♦ 

12 

IMP 

Integrated  Master  Plan 

IMP 

Integrated  Master  P 

an 

Integrated  Master  Schedule 
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Assessing  SE  Capability 
Data  Analysis  Phase 


Synthesize  Data  &  Identify 
Strengths  &  Weaknesses 


Score  Capabilities 


Opining  jO  05  Wtiil  neat  of  ifKtems  ■r<gin««iirig  *it  you  molvtd  I 


SukshoMer  KM  03  Can  you  provuto  *n  ca. 

Rftqi.r*mi*rtf*  lit.ik»li£ild«f'c  t*q.,rc.m»r,H  • 
ll»»r»(iur-  |«ni)  ligw  IM  wrs  :c«>ted  !■' 


Interview  Data 


jir/  Ut  Atttii  ly| 

».  ill 

|*.«ptui»<l  «i  «  n 


yp»*  Ti  iftivj'iftiftSiil?  (lur^lir*!*1 
LftplUtftll  «l  ft  tUUftxIftlll  fllftlllHW'* 


D*ngi 

I  Implvfln. 


1 

'•guiwmtntf  «ny  t  !i*iftad^hc*l  fht  n*w  u«r  r«qi*»»m**iu  Tta 
■lalftlift-'1  -I  ftAft! !»--! 


Q  #  Document  Review  Notes 


P  Spec 


SEP 


2.01 


TEMP 


Performance  Specification 


Systems  Engineering  Plan 


'The  established  reqmtsi  database  shall  be 
the  autoritative  requirements  management 
tool  for  capture,  recording  and  to  provide 
full  traceability." 


c> 


ICD 


TP 

IMP 


|  Document  Reviews  Data 


[Integrated  Master  Plan 


Category 


01.01  Who  are  the  stakeholders  for  the 
i^stem^ 


01.02  Do  stakeholders  feel  that  their 
requirements  are  captured  effectively? 


01.07  How  are  new  requirements  or 
changes  in  requirements  vetted  for 
approval? 


Stakeholder 

Requirements 

Definition 


01.04  How  are  requirements  problems, 
such  as  conflicting  stakeholder 
requirements,  identified  and  resolved? 


01.05  Is  there  a  formally  documented 
and  approved  Concept  of  Operations? 


Requirements 

Analysis 


Question 


ezi 

f 

- 


Criteria: 


itakeholders  are  identified  and  understood  across  the 

ir°qram 


Justification: 


Stakeholders  were  identified  in  the  Acquisition  Plan,  £ 
and  Communication  Plan  consistently 


requirements  and  i 
for  approval 


01.08  **  How  are  stakeholder  desires  for 
interfaces  and  interoperability  with 
external  systems  captured? 


01.06  Are  methods  such  as  use  cases, 
mission  threads,  etc.  used  to  help 
develop  and  derive  requirements?  Are 
cases  developed  in  coordination 
with  the  system  architects? 


02.02  What  tool  is  used  for  storing  and 
managing  requirements? 


02.06  Can  you  provide  an  example  of 
how  poorly  defined  requirements  (e.g., 
-testable,  poorly  defined)  have  been 
identified  and  resolved? 


02.01  Are  all  types  of  requirements 
(functional,  performance,  design 
constraints,  regulatory,  etc)  captured  in 
consistent  manner? 


02.05  Can  you  give  an  example  of  any 
'equipments  that  do  not  have  top- 
downJraceabiMt^?^^^^^_^^^^^_ 


I 


Requirements  for  interf. 
extemal s^stemsjjrej1o 


Used  cases  (scenarios) 
recjujrements^^^^ 


Fully  Capable 

Strong  Capability 

Moderate  Capability 

Weak  Capability 

No  Capability 

Reauirements  are  caotured  in  consistent  format 

Limited  evidence  of  this 

Lower  level  requirements  are  all  traceable  to  higher  level 
reauirements 

No  traceabilitv  reported 

Analyze  Underlying  Dimensions  of  Capability 


2.0  Requirements  Analysis  Findings  (Notional) 

Governance 

While  the  program  SEP  calls  for  use  of  a  req  mgmt  tool  to  manage 
requirements  no  governance  mechanisms  are  in  place  for  oversight. 
Requirements  are  changed  without  notifying  key  stakeholders 

People 

Some  individuals  who  need  access  to  the  latest  requirements  on 
programs  do  not  know  how  to  access  or  use  the  tools. 

Process 

No  formal  overarching  requirements  management  process  was 
identified,  team  members  create  ad  hoc  methods  across  programs  and 
do  not  follow  processes  within  program  SEPs. 

The  Requirements  Management  tools  available  to  the  team  are 
comprehensive  and  no  issues  with  access  for  those  trained  in  use  of 
the  tool 

[ 


Document  Findings 


Interview  Summary 
Cross  Reference 
Underlying  Issues 


Formulate 
Recommendations 


Based  upon  underlying 
dimensions,  capability 
interdependencies  & 
characterize  impact 
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Assessment  Results  & 
SEI/NDA  Study  Findings 


Plot  of  NDIA-SEI SE  Effectiveness  Study  & 
Notional  Independent  Assessment  Findings 


Requirements  Mgmt 


Risk  Management 


K  A 

^ - 

Product  Integration 


Lowest 

Priority 


1.0 


2.0 


3.0 


4.0 


5.0 


Composite  Ranking 


Risk  Management 
Requirements  Mgmt 
Project  Planning 
Trade  Studies 


Product  Integration 


Plot  provides  interesting 
insight  into  rankings, 
however  other  factors 
must  be  considered  for 
prioritization 

Underlying  causal  factors  from 
capability  dimensions  of  People 
Process,  Technology  and 
Governance 

Balance  of  organizational  risks 
and  trades  to  optimize  ROI 


Scorecard 

Rank 


C 

tx 

CD 

ro 

CL 

V) 

hi 

0 

DC 

Q) 

C 

Composite  Scorecard  Value 
from  Assessment 


Project,  program  or  portfolio 
Phase(s),  Schedule(s),  Funding 
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Prioritized  Systems  Engineering 
Capability  Development  Plan 


TH/TDEI 


The  Capability  Development  Plan: 


Leverages  data  and  actual  performance 
from  the  diagnostic  to  create  tangible  and 
actionable  recommendations 


Hones  in  on  underlying  causes  providing 
synergy  in  improvement  efforts  for 
greatest  Return  on  Investment  (ROI) 


Accounts  for  interdependencies  between 
capabilities  and  provides  necessary 
insight  to  prioritize  efforts  for  rapid  and 
immediate  impact 


Lays  our  the  necessary  prioritized  tasks 
Is  a  detailed  and  prioritized  work  plan 


I 


Plan  creates  a  catalyst  for  change  to 
institutionalize  Systems  Engineering 


Findings  & 
Recommendations 


li. 


r\ 


Underlying  Dimensions 


2.0  Requirements  Analysis  Findings  (Notional) 

Governance 

While  the  program  SEP  callsforuse  of  DOORs  to  manage  requirements 
ro  governaoe  mechanisms  are  in  place  for  oversight.  Requirements  are 
shanged  without  notif  ing  key  stakeholders 

People 

Some  indiu duals  who  need  access  to  the  latest  requirements  cn 
arog-ams  do  not  know  howto  access  or  use  the  tods. 

La 

No  formal  overarch ng  requirements  management  process  was 
dentified.team  members  create  ad  hoc  methods  across  programs  and 
|do  not  follow  processes  within  program  SEPs. 

[The  Requirements  Management  tools  available  to  the  team  are 
comprehensive  and  no  issues  with  access  for  those  trained  in  use  of 
fhe  tool 

Core  &  Supporting 
SE  Capabilities 


Prioritized  Plan 


1.  Requirements  Mgmt 

2.  Risk  Management 

3.  Project  Planning 


4.  Product  Integration 


4.  Trade  Studies 


Plan  &  Schedule 


r 

% 

ITr 

1 

EfETL, 

J  1  H 

> 


Systems  Engineering 
Capability  Development  Plan 


MA 

Tank  Automotive  Research  Development  &  Engineering  Command 


October  14.  2008 


I  /  ■ 
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Summary 


TFinSDtEC 


Premise: 

SE  Capability 


Program  Performance 


SE  Capability  is  arguably  one  of  the  most  important  for 
companies  that  develop  and  integrate  complex  systems 


Challenges 

Typically  Seen  in  Organizations 


* 


Benefits 


Building  a  comprehensive  view  of  capability 
with  an  understanding  of  interdependencies 
to  create  a  high  performing  organization 


11 

m 

•"I . 

is 

sssssr; 

Integrated  Acquisition  Capability  a  comprehensive 
framework  to  assess  and  build  the  capabilities  essential 
for  a  successful  system  acquisition  program 


Obtaining  unhindered  and  unbiased  feedback 
and  applying  a  proven  approach  for 
improvement 


Tailored,  independent  and  objective  review  based 
upon  industry  standards  and  best  practices.  Dual  path 
(two-way)  verification  ensures  integrity  of  results 


Leverage  resources  to  implement 
improvement  efforts  in  lieu  of  core  mission  and 
Identifying  key  areas  to  improve  performance 


Governance 


People 


Process 


Technology  & 
Infrastructure 


Diagnostic  identifies  underlying  causes  of  capability 
inhibitors  and  offers  insight  to  provide  rapid  and 
synergistic  improvements 


Establishing  a  concrete  baseline  from  which 
to  measure  performance  to  appropriately 
adapt  make  course  corrections 


Fully  Capable 
Some  Capability 
Limited  Capability 
Very  Little  Capability 
No  Capability 


Identifies  improvement  opportunities  &  strengths  to 
leverage.  Creates  a  Current  State  Baseline  from  which 
to  track  improvement. 


Breaking  down  organizational  barriers  and 
building  integrated  capabilities 


Prioritized  plan  provides  realistic  and  tangible 
recommendations  and  creates  a  catalyst  for  change  to 
institutionalize  Systems  Engineering 


Conclusion: 

Approach  enables  SE  Maturation  for  Increased  Program  Performance 
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Intro 


■  This  briefing  was  developed  during  funded  research  for  the 
U.  S.  Air  Force  Aeronautical  Systems  Center  for  the  AEA 
Capability  Planning  Manager  (ASC/XRS) 

■  This  briefing  is  unclassified  in  its  entirety 
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Purpose  Statement 

■  Discuss  the  methodology  to  build  an  adaptable  System  of  Systems 
architecture  that  can  be  used  to  compare  performance  of  alternative 
solutions. 

■  Definitions 

"  Adaptable  -  capable  of  becoming  suitable  to  a  particular  situation  or  use 

■  System  of  Systems  -  a  set  or  arrangement  of  systems  that  results  when  independent 
and  useful  systems  are  integrated  into  a  larger  system  that  delivers  unique  capabilities 
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Outline 

■  AEA  SoS  Description 

■  Focus  of  Effort 

■  Methodology 

■  Architecture  Challenges 

■  Solutions 

■  System  Analyses 
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Airborne  Electronic  Attack  System  of  Systems 
(AEA  SoS)  Description 

■  Limited  number  of  AEA  assets  support  multiple  air  and  ground  elements  against  multiple 
threats 

■  Requires  informed  AEA  decisions  across  the  theater  in  real-time 

■  Requires  coordination  between  a  variety  of  assets  (SoS)  to  improve: 

■  AEA  tasking  awareness 

■  Flexibility  and  confidence  to  make  changes 

■  Overall  AEA  Efficiency 

■  Goal  -  to  improve  AEA  support  through  interoperability  &  coordination 

■  Information  sharing 

■  Management  of  assets 
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Focus  of  Effort 

■  Develop  a  means  to  verify  that  the  SoS  provides  significant  improvements 
to  combat  effectiveness 

■  Develop  a  means  to  quantify  those  improvements 

■  Determine  which  ‘attributes’  make  a  statistically  significant  difference 
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Methodology 


■  Build  an  adaptable  architecture  to  model  the  AEA  SoS 

■  Using  the  architecture  as  a  baseline,  perform  Systems  Analyses  to 
determine  and  measure  the  improvements  to  combat  effectiveness 

■  Screening  model  -  to  identify  the  key  ‘attributes’ 

■  High  Fidelity  model  -  to  determine  effectiveness 
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Architecture  Challenges 

■  Need  an  adaptable  architecture  that  represents  various: 

■  Configurations 

■  Situations 

■  Attributes 
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Architecture  Challenge  -  Various  Configurations 


■  AEA  SoS  Architecture  must  be  adaptable  to  many  different  configurations 

■  AEA  SoS  consists  of  many  different  players/roles 

■  AEA  Platforms  (Jammers) 

■  Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  Platforms 

■  Protected  Element  (Bombers,  Ground  troops,  etc) 

■  Command  Element  (Air  Operations  Center,  Air  Control  aircraft,  etc) 

■  AEA  Battle  Management  (Operational-level,  Tactical-level) 

■  Each  role  can  be  thought  of  as  its  own  Family  of  Systems 


■  Definition 

■  Family  of  Systems  -  a  set  of  systems  that  provide  similar  capabilities  through  different  approaches 
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Solution  -  Generic  Activity  Modeling 

■  Activity  diagrams  -  used  to  model  activities  and  exchanges  within  the 
AEA  SoS 

■  Abstract  Operational  Node  classes  -  defined  to  account  for  variable 
configurations 

■  Abstract  High  Level  Activities  -  defined  for  each  operational  node 

■  Abstract  Information  Element  classes  -  defined  to  represent  the 
information  exchanges  between  operational  node  activities 

■  Result  -  an  all-encompassing  “one  size  fits  all”  operational  model 

■  Definitions 

"  Generic  -  very  comprehensive,  relating  to  or  descriptive  of  an  entire  group  or  class 

■  Abstract  -  thought  of  or  stated  without  reference  to  a  specific  instance;  generalized 
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Architecture  Challenge  -  Various  Situations 


October  26,  2006 


■  AEA  SoS  Architecture  must  be  adaptable  to  the  many  different 
‘situations’  that  may  occur  during  a  mission 

■  New  Jamming  Request  from  the  Protected  Element 

■  AEA  Platform  Malfunction 

■  Change  in  Mission  Priorities 

■  Command  Element  Cancels  Mission 

■  React  to  a  Pop-up  SAM 
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Solution  -  Notional  Modeling  of  Specific  Situations 

■  Activity  diagrams  -  used  to  model  specific  ‘situations’ 

■  Derived  from  notional  Execute  AEA  Mission  Activity  Diagram 

■  Each  Situation  represents  a  single  thread  through  the  architecture 
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Solution  -  Notional  Modeling  of  Specific  Situations 


slide  14 


Approved  for  public  release,  Case  Number  88ABW-2008-0319,  29  Sep  08 


October  26,2006 


Architecture  Challenge  -  Various  Attributes 


■  The  AEA  SoS  Architecture  must  be  adaptable  to  take  into  account  a 
number  of  various  ‘attributes’  that  can  change  from  one  mission  to  the 
next. 


■  Some  examples  out  over  40  identified  attributes: 

■  AEA  -  PE  Support  Relationship 

■  Communications  Quality 

■  Jammer  Effectiveness 
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Using  the  adaptable  architecture 


Method: 

1 .  For  each  swimlane,  show  settings  for 
appropriate  attributes 

2.  Inside  each  swimlane,  show  standardized 
operations  functions 

3.  Build  multiple  configurations  (attributes  & 
functions) 

4.  Model  attribute  and  function  interactions 
using  the  architecture  foundation 

5.  Simulate  to  compare  performance  from 
different  configurations 


AEA  Operator 

Cognitive  attributes 

Information  attributes 

Physical  attributes 

Maneuver 

Sense 

Communicate 

Process 

Engage 


Developed  from  SV 
Functional  Areas 


Functions 
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AEA  Objects 

Cognitive  attributes 
Support  Relations 


Information  attributes 
Comms  quality 

Physical  attributes 
AEA  Jammer 


AEA  Operator 

V  — - 

AEA  1 

AEA  2 

AEA  3 

None  Direct  Close  TACON 

None  Direct 

Close  TACON 

None  Direct  Close  TACON 

■■■ 

1 

Degraded  Nominal  Perfect 

_d  _^i  _lI 

Degraded  Nominal  Perfect 

_d 

Degraded  Nominal  Perfect 

_il 

t 

o 

o 

o 

■ 

Stand  Off  Jammer 


i 


Stand  In  Jammer  N  Escort  Jammer 


■ 


Maneuver 

Maneuver 

Maneuver 

Sense 

Sense 

Sense 

Communicate 

Communicate 

Communicate 

Process 

Process 

Process 

Engage 

Engage 

Engage 

Functions 
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2.  Single 
Configuration 

(example) 


Protected 
Entity 

Objects 


Swim  lanes  (Roles) 


Bomber 


Attributes 


Functions 


M,  s,  C,  P,  E 


Adversary 


Radar 


M,  S,  C,  P,  E 


Air  Recon 


M,  s,  C,  P 


Oper 


Tac 

AEA  1 

AEA  2 

AEA  3 

Air  Control 

c,p 


M,  S,  C,  P 


AEA  Operator 


SOJ 

L 


A  A  A 

su 


Escort 

t 


M,  S,  C,  P,  E 


M,  S,  P,  E 


M,  S,  C,  P,  E 


Command  Element 


CAOC 


M,  S,  C,  P 


C,P 
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3.  Multiple 
configurations 


■  Each  configuration  accounts  for  all 
swim  lanes  &  functions 

■  Each  configuration  has  different: 

■  Attributes 

■  Cognitive  /  authorities 

"  Information  /  communications 

■  Physical  /  platform  types 

■  Functions 

■  Attribute  impacts  on 
performance 


him-M!  TOTH  hill  CUKUCWm  *  TFI  TPHjILrfc^Y 
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Configuration  C 
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4.  Attributes  impact 
on  functions 


Nominal  values  shown.  Simulations 
calculations  generated  from  Triangle 
distributions  (Lowest,  Nominal,  Highest) 


2% 

0°/q/ 

Cognitive 

AEA-PE  Support 
Relationship 


Informational 
Comms  Quality 


Spatial 

relationships 


Velocity  and 
acceleration 
data 


Sensor 

interpretation 


Sensor 

data/reports 


Message  interpretation 
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Nominal 

Perfect 


Speed 

90  sec 
43  sec 
0 


None 

Direct 

Close 

TACON 


Speed 
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+2% 

+5% 

+5% 


Algorithms 


Weapon  control 


Weapon  data 


Physical 

Effectiveness 


Platform 

characteristics 


Sensor 

characteristics 


Radio/Data  Link  characteristics 


Computer  characteristics 


Maneuver 


Sense 


Communicate 


Process 


■  Effectiveness 

■  Effectiveness 
Error 

■  Jammer 
Location 


Engage 
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Functions  from  the  architecture’s  System  Views  (SV)  NOTIONAL  Data 
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Simulation  Courses  of  Action  (COA) 
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5.  Simulate  to  compare  performance  from  different 
configurations 


■  Course  Of  Action  (COA)  Scorer  model 

■  Jammer  location 

■  Expected  Jammer  Effectiveness 

■  Time  to  implement 


■Monte  Carlo  Simulation 

■Attributes’  effect  on  Battle 
Manager’s  Decision  Window 

Do  longer  decision  windows  make  a  difference  in 
AEA  combat? 

For  these  configurations,  faster  decisions 
increased  jammer  effectiveness  by  45%  and  53% 


5THFNGTH  THUM'dl  CMJCWTTl  A  TFI  TfVJLJS^V 


Notional  Jammer  Effectiveness 
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Scenario  Summary 
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Result  Cells: 

Jammer  effectiveness  163.3  118.3  110.0 


Less  is  better 
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5.  Simulate  to  compare  performance  from  different 
configurations 


Sample  data  plots  using  JMP  ANOVA 


NOTIONAL  Data 

- API 
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■  Adaptable  Architecture  provides  a  neutral  arena  to  compare  performance 
from  multiple  alternatives 

■  AA  employs  a  capability-based  approach  vs  platform-based  approach  to  SoS 
solutions 


■  AA  enables  a  comprehensive  analysis  across  different  force  configurations 
and  dynamic  situations 
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Incremental  Commitment  Model  (ICM): 

Nature  and  Origins 


•  Integrates  hardware,  software,  and  human  factors  elements  of 
systems  engineering 

-  Concurrent  exploration  of  needs  and  opportunities 

-  Concurrent  engineering  of  hardware,  software,  human  aspects 

-  Concurrency  stabilized  via  anchor  point  milestones 

•  Developed  in  response  to  DoD-related  issues 

-  Clarify  “spiral  development”  usage  in  DoD  Instruction  5000.2 

•  Initial  phased  version  (2005) 

-  Explain  Future  Combat  System  of  systems  spiral  usage  to  GAO 

•  Underlying  process  principles  (2006) 

-  Provide  framework  for  human-systems  integration 

•  National  Research  Council  report  (2007) 

•  Integrates  strengths  of  current  process  models 

-  But  not  their  weaknesses 
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ICM  integrates  strengths  of  current  process  models 

But  not  their  weaknesses 

*  V-Model:  Emphasis  on  early  verification  and  validation 

-  But  not  ease  of  sequential,  single-increment  interpretation 

*  Spiral  Model:  Risk-driven  activity  prioritization 

-  But  not  lack  of  well-defined  in-process  milestones 

*  RUP  and  MBASE:  Concurrent  engineering  stabilized  by 
anchor  point  milestones 

-  But  not  software  orientation 

*  Lean  Development:  Emphasis  on  value-adding  activities 

-  But  not  repeatable  manufacturing  orientation 

*  Agile  Methods:  Adaptability  to  unexpected  change 

-  But  not  software  orientation,  lack  of  scalability 
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Process  Model  Principles 

Principles  trump  diagrams 


1.  Commitment  and  accountability 

2.  Success-critical  stakeholder  satisficing 

3.  Incremental  growth  of  system  definition  and 
stakeholder  commitment 


4,  5.  Concurrent,  iterative  system  definition  and 
development  cycles 

Cycles  can  be  viewed  as  sequential  concurrently- 
performed  phases  or  spiral  growth  of  system 
definition 

6.  Risk-based  activity  levels  and  anchor  point 
commitment  milestones 


Used  by  60-80%  of  CrossTalk  Top-5  projects,  2002-2005 
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Common  Risk-Driven  Special  Cases  of  the  ICM 


Special  Case 


Example 


Size, 

Complexity 


Change 
Rate  % 
/Month 


Criticality 


NDI  Support 


Org,  Personnel 
Capability 


Key  Stage  I  Activities  :  Incremental  Definition 


Key  Stage  II  Activities:  Incremental 
Development,  Operations 


Time  per  Build; 
per  Increment 


1.  Use  NDI 

Small  Accounting 

Complete 

Acquire  NDI 

Use  NDI 

2.  Agile 

E-services 

Low 

1-30 

Low-Med 

Good; 
in  place 

Agile-ready 

Med-high 

Skip  Valuation  ,  Architecting  phases 

Scrum  plus  agile  methods  of  choice 

<=  1  day; 

2-6  weeks 

3.  Architected 

Agile 

Business  data 
processing 

Med 

1-10 

Med-High 

Good; 

most  in  place 

Agile-ready 

Med-high 

Combine  Valuation,  Architecting 
phases.  Complete  NDI  preparation 

Architecture-based  Scrum  of  Scrums 

2-4  weeks; 

2-6  months 

4.  Formal  Methods 

Security  kernel; 
Safety-critical  LSI 
chip 

Low 

0.3 

Extra  High 

None 

Strong  formal 

methods 

experience 

Precise  formal  specification 

Formally-based  programming 
language;  formal  verification 

1-5  days; 

1  -4  weeks 

5.  HW  component 
with  embedded 

SW 

Multi-sensor 
control  device 

Low 

0.3-1 

Med-Very 

High 

Good; 

In  place 

Experienced; 

med-high 

Concurrent  HW/SW  engineering.  CDR- 
level  ICM  DCR 

IOC  Development,  LRIP,  FRP. 

Concurrent  Version  N+1  engineering 

SW:  1-5  days; 
Market-driven 

6.  Indivisible  IOC 

Complete  vehicle 
platform 

Med- 

High 

0.3-1 

High-Very 

High 

Some  in  place 

Experienced; 

med-high 

Determine  minimum-IOC  likely, 
conservative  cost.  Add  deferrable  SW 
features  as  risk  reserve 

Drop  deferrable  features  to  meet 
conservative  cost.  Strong  award  fee 
for  features  not  dropped 

SW:  2-6  weeks; 
Platform:  6-18 
months 

7.  NDI-  Intensive 

Supply  Chain 
Management 

Med- 

High 

0.3-3 

Med-Very 

High 

NDI-driven 

architecture 

NDI-experienced; 

Med-high 

Thorough  NDI-suite  life  cycle  cost- 
benefit  analysis,  selection,  concurrent 
requirements/  architecture  definition 

Pro-active  NDI  evolution  influencing, 

NDI  upgrade  synchronization 

SW:  1-4  weeks; 
System:  6-18 
months 

9.  Hybrid  agile/ 

plan-driven 

system 

C4ISR 

Med- 
Very  High 

Mixed 

parts: 

1-10 

Mixed  parts; 
Med-Very 

High 

Mixed  parts 

Mixed  parts 

Full  ICM;  encapsulated  agile  in  high 
change,  low-medium  criticality  parts 
(Often  HMI,  external  interfaces) 

Full  ICM  , three-team  incremental 
development,  concurrent  V&V,  next- 
increment  rebaselining 

1-2  months; 

9-18  months 

9.  Multi-owner 
system  of  systems 

Net-centric 
military  operations 

Very  High 

Mixed 

parts: 

1-10 

Very  High 

Many  NDIs; 
some  in  place 

Related 

experience,  med- 
high 

Full  ICM;  extensive  multi-owner  team 
building,  negotiation 

Full  ICM;  large  ongoing 
system/software  engineering  effort 

2-4  months;  18- 
24  months 

10.  Family  of 
systems 

Medical  Device 
Product  Line 

Med- 
Very  High 

1-3 

Med  -  Very 

High 

Some  in  place 

Related 

experience,  med 
-  high 

Full  ICM;  Full  stakeholder  participation 
in  product  line  scoping.  Strong  business 
case 

Full  ICM.  Extra  resources  for  first 
system,  version  control,  multi¬ 
stakeholder  support 

1-2  months;  9- 
18  months 

C4ISR:  Command,  Control,  Computing,  Communications,  Intelligence,  Surveillance,  Reconnaissance.  CDR:  Critical  Design  Review. 
DCR:  Development  Commitment  Review.  FRP:  Full-Rate  Production.  HMI:  Human-Machine  Interface.  HW:  Hard  ware. 

IOC:  Initial  Operational  Capability.  LRIP:  Low-Rate  Initial  Production.  NDI:  Non-Development  Item.  SW:  Software 
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Casel:  Use  NDI 


*  Exploration  phase  identifies  NDI  opportunities 

•  NDI  risk/opportunity  analysis  indicates  risks  acceptable 

-  Product  growth  envelope  fits  within  NDI  capability 

-  Compatible  NDI  and  product  evolution  paths 

-  Acceptable  NDI  volatility,  some  open-source  components  highly  volatile 

-  Acceptable  usability,  dependability,  interoperability 

-  NDI  available  or  affordable 

•  Example:  Small  accounting  system 

*  Size/complexity:  Low 

•  Anticipated  change  rate  (%  per  month):  Low 

*  Criticality:  Low 

•  NDI  support:  Complete 

*  Organization  and  personnel  capability:  NDI-experienced 

*  Key  Stage  I  activities:  Acquire  NDI 

*  Key  State  II  activities:  Use  NDI 

•  Time/build:  Driven  by  time  to  initialize/tailor  NDI 

•  Time/increment:  Driven  by  NDI  upgrades 
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Case  2:  Pure  Agile  Methods 


•  Exploration  phase  determines 

-  Low  product  and  project  size  and  complexity 

-  Fixing  increment  defects  in  next  increment  acceptable 

-  Existing  hardware  and  NDI  support  of  growth  envelope 

-  Sufficient  agile-capable  personnel 

-  Need  to  accommodate  rapid  change,  emergent  requirements,  early  user  capability 

•  Example:  E-services 

•  Size/complexity:  Low 

•  Anticipated  change  rate  (%  per  month):  1-30% 

•  Criticality:  Low  to  medium 

•  NDI  support:  Good;  in  place 

•  Organization  and  personnel  capability:  Agile-ready,  medium  to  high 
capability 

•  Key  Stage  I  activities:  Skip  Valuation  and  Architecting  phases 

•  Key  State  II  activities:  Scrum  plus  agile  methods  of  choice 

•  Time/build:  Daily 

•  Time/increment:  2-6  weeks 
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Case  3:  Architected  Agile 


•  Exploration  phase  determines 

-  Need  to  accommodate  fairly  rapid  change,  emergent  requirements,  early  user 
capability 

-  Low  risk  of  scalability  up  to  100  people 

-  NDI  support  of  growth  envelope 

-  Nucleus  of  highly  agile-capable  personnel 

-  Moderate  to  high  loss  due  to  increment  defects 

•  Example:  Business  data  processing 

•  Size/complexity:  Medium 

•  Anticipated  change  rate  (%  per  month):  1-10% 

•  Criticality:  Medium  to  high 

•  NDI  support:  Good,  most  in  place 

•  Organization  and  personnel  capability:  Agile-ready,  med-high  capability 

•  Key  Stage  I  activities:  Combined  Valuation  and  Architecting  phase, 
complete  NDI  preparation 

•  Key  State  II  activities:  Architecture-based  scrum  of  scrums 

•  Time/build:  2-4  weeks  Time/increment:  2-6  months 
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Case  4:  Formal  Methods 


•  Biggest  risks:  Software/hardware  does  not  accurately  implement 
required  algorithm  precision,  security,  safety  mechanisms,  or 
critical  timing 

•  Example:  Security  kernel  or  safety-critical  LSI  chip 

•  Size/complexity:  Low 

•  Anticipated  change  rate  (%  per  month):  0.3% 

•  Criticality:  Extra  high 

•  NDI  support:  None 

•  Organization  and  personnel  capability:  Strong  formal  methods 
experience 

•  Key  Stage  I  activities:  Precise  formal  specification 

•  Key  State  II  activities:  Formally-based  programming  language; 
formal  verification 

•  Time/build:  1-5  days 

•  Time/increment:  1-4  weeks 
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Case  5:  Hardware  Component  with 
Embedded  Software 


•  Biggest  risks:  Device  recall,  lawsuits,  production  line  rework,  hardware- 
software  integration 

-  DCR  carried  to  Critical  Design  Review  level 

-  Concurrent  hardware -software  design 

*  Criticality  makes  Agile  too  risky 

-  Continuous  hardware-software  integration 

•  Initially  with  simulated  hardware 

•  Low  risk  of  overrun 

-  Low  complexity,  stable  requirements  and  NDI 

-  Little  need  for  risk  reserve 

-  Likely  single-supplier  software 
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Case  5:  Hardware  Component  with 

Embedded  Software  (continued) 


•  Example:  Multi-sensor  control  device 

•  Size/complexity:  Low 

•  Anticipated  change  rate  (%  per  month):  0.3-1% 

•  Criticality:  Medium  to  very  high 

•  NDI  support:  Good,  in  place 

•  Organization  and  personnel  capability:  Experienced;  medium  to  high 
capability 

•  Key  Stage  I  activities:  Concurrent  hardware  and  software  engineering; 
CDR-level  ICM  DCR 

•  Key  State  II  activities:  IOC  Development,  LRIP,FRP,  concurrent  version 
N+1  engineering 

•  Time/build:  1 -5  days  (software) 

•  Time/increment:  Market-driven 
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Case  6:  Indivisible  IOC 

•  Biggest  risk:  Complexity,  NDI  uncertainties  cause  cost-schedule 
overrun 

-  Similar  strategies  to  case  4  for  criticality  (CDR,  concurrent  HW- 
SW  design,  continuous  integration) 

-  Add  deferrable  software  features  as  risk  reserve 

•  Adopt  conservative  (90%  sure)  cost  and  schedule 

•  Drop  software  features  to  meet  cost  and  schedule 

•  Strong  award  fee  for  features  not  dropped 

-  Likely  multiple-supplier  software  makes  longer  (multi-weekly) 
builds  more  necessary 
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Case  6:  Indivisible  IOC  (continued) 


•  Example:  Complete  vehicle  platform 

•  Size/complexity:  Medium  to  high 

•  Anticipated  change  rate  (%  per  month):  0.3-1% 

•  Criticality:  High  to  very  high 

•  NDI  support:  Some  in  place 

•  Organization  and  personnel  capability:  Experienced,  medium  to 
high  capability 

•  Key  Stage  I  activities:  Determine  minimum-IOC  likely,  conservative 
cost;  Add  deferrable  software  features  as  risk  reserve 

•  Key  State  II  activities:  Drop  deferrable  features  to  meet 
conservative  cost;  Strong  award  fee  for  features  not  dropped 

•  Time/build:  2-6  weeks  (software) 

•  Time/increment:  6-18  months  (platform) 
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Case  7:  NDI-Intensive 


•  Biggest  risks:  incompatible  NDI;  rapid  change,  business/mission 
criticality;  low  NDI  assessment  and  integration  experience;  supply  chain 
stakeholder  incompatibilities 

•  Example:  Supply  chain  management 

•  Size/complexity:  Medium  to  high 

•  Anticipated  change  rate  (%  per  month):  0.3-3% 

•  Criticality:  Medium  to  very  high 

•  NDI  support:  NDI-driven  architecture 

•  Organization  and  personnel  capability:  NDI-experienced;  medium  to 
high  capability 

•  Key  Stage  I  activities:  Thorough  NDI-suite  life  cycle  cost-benefit 
analysis,  selection,  concurrent  requirements  and  architecture  definition 

•  Key  State  II  activities:  Pro-active  NDI  evolution  influencing,  NDI  upgrade 
synchronization 

•  Time/build:  1 -4  weeks  (software) 

•  Time/increment:  6-18  months  (systems) 
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Case  8:  Hybrid  Agile/Plan-Driven  System 


•  Biggest  risks:  large  scale,  high  complexity,  rapid  change,  mixed 
high/low  criticality,  partial  NDI  support,  mixed  personnel  capability 

•  Example:  C4ISR  system 

•  Size/complexity:  Medium  to  very  high 

•  Anticipated  change  rate  (%  per  month):  Mixed  parts;  1-10% 

•  Criticality:  Mixed  parts;  medium  to  very  high 

•  NDI  support:  Mixed  parts 

•  Organization  and  personnel  capability:  Mixed  parts 

•  Key  Stage  I  activities:  Full  ICM;  encapsulated  agile  in  high 
changed;  low-medium  criticality  parts  (often  HMI,  external 
interfaces) 

•  Key  State  II  activities:  Full  ICM,  three-team  incremental 
development,  concurrent  V&V,  next-increment  rebaselining 

•  Time/build:  1-2  months 

•  Time/increment:  9-18  months 
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Case  9:  Multi-Owner  System  of  Systems 


•  Biggest  risks:  all  those  of  Case  8  plus 

-  Need  to  synchronize,  integrate  separately-managed,  independently-evolving 
systems 

-  Extremely  large-scale;  deep  supplier  hierarchies 

-  Rapid  adaptation  to  change  extremely  difficult 

•  Example:  Net-centric  military  operations 

*  Size/complexity:  Very  high 

•  Anticipated  change  rate  (%  per  month):  Mixed  parts;  1-10% 

*  Criticality:  Very  high 

•  NDI  support:  Many  NDIs;  some  in  place 

*  Organization  and  personnel  capability:  Related  experience,  medium  to 
high 

*  Key  Stage  I  activities:  Full  ICM;  extensive  multi-owner  teambuilding, 
negotiation 

*  Key  State  II  activities:  Full  ICM;  large  ongoing  system/software 
engineering  effort 

•  Time/build:  2-4  months  Time/increment:  1 8-24  months 
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Case  10:  Family  of  Systems 


•  Biggest  risks:  all  those  of  Case  8  plus 

-  Need  to  synchronize,  integrate  separately-managed,  independently- 
evolving  systems 

-  Extremely  large-scale;  deep  supplier  hierarchies 

-  Rapid  adaptation  to  change  extremely  difficult 

•  Example:  Medical  device  product  line 

•  Size/complexity:  Medium  to  very  high 

•  Anticipated  change  rate  (%  per  month):  1-3% 

•  Criticality:  Medium  to  very  high 

•  NDI  support:  Some  in  place 

•  Organization  and  personnel  capability:  Related  experience,  medium 
to  high  capability 

•  Key  Stage  I  activities:  Full  ICM;  full  stakeholder  participation  in 
product  line  scoping;  strong  business  case 

•  Key  State  II  activities:  Full  ICM;  extra  resources  for  first  system, 
version  control,  multi-stakeholder  support 

•  Time/build:  1-2  months  Time/increment:  9-18  months 
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Frequently  Asked  Question 


Q:  Having  all  that  ICM  generality  and  then  using  the  decision 
table  to  come  back  to  a  simple  model  seems  like  an  overkill. 

-  If  my  risk  patterns  are  stable,  can’t  I  just  use  the  special  case 
indicated  by  the  decision  table? 


A:  Yes,  you  can  and  should  -  as  long  as  your  risk  patterns  stay 
stable.  But  as  you  encounter  change,  the  ICM  helps  you  adapt  to 
it. 

-  And  it  helps  you  collaborate  with  other  organizations  that  may  use 
different  special  cases. 
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Using  the  Incremental  Commitment 
Model  (ICM)  to  Help  Execute 
Competitive  Prototyping  (CP) 

— Charts  with  Notes — 


Barry  Boehm  and  Jo  Ann  Lane 
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Center  for  Systems  and  Software  Engineering 
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Motivation  and  Context 


•  DoD  is  emphasizing  CP  for  system  acquisition 

-  Young  memo,  September  2007 

•  CP  can  produce  significant  benefits,  but  also 
has  risks 

-  Benefits  related  to  incremental  commitment 

-  Examples  of  risks  from  experiences,  workshops 

•  The  risk-driven  ICM  can  help  address  the  risks 

-  Primarily  through  its  underlying  principles 
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Young  Memo:  Prototyping  and  Competition 


*  Discover  issues  before  costly  SDD  phase 

-  Producing  detailed  designs  in  SDD 

-  Not  solving  myriad  technical  issues 

*  Services  and  Agencies  to  produce  competitive 
prototypes  through  Milestone  B 

-  Reduce  technical  risk,  validate  designs  and  cost  estimates, 
evaluate  manufacturing  processes,  refine  requirements 

*  Will  reduce  time  to  fielding 

-  And  enhance  govt.-industry  teambuilding,  SysE  skills, 
attractiveness  to  next  generation  of  technologists 

*  Applies  to  all  programs  requiring  USD(AT&L)  approval 

-  Should  be  extended  to  appropriate  programs  below  ACAT  I 
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Incremental  Commitment  in 

Gambling 


*  Total  Commitment:  Roulette 

-  Put  your  chips  on  a  number 

*  E.g.,  a  value  of  a  key  performance  parameter 

-  Wait  and  see  if  you  win  or  lose 

•  Incremental  Commitment:  Poker,  Blackjack 

-  Put  some  chips  in 

-  See  your  cards,  some  of  others’  cards 

-  Decide  whether,  how  much  to  commit  to 
proceed 


03/19/2008 


©USC-CSSE 


5 


III|E| 


University  of  Southern  California 

Center  for  Systems  and  Software  Engineering 


Scalable  remotely  controlled 

operations 
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Total  vs.  Incremental  Commitment -4:1  RPV 

•  Total  Commitment 

-  Agent  technology  demo  and  PR:  Can  do  4:1  for  $1B 

-  Winning  bidder:  $800M;  PDR  in  120  days;  4:1  capability  in  40  months 

-  PDR:  many  outstanding  risks,  undefined  interfaces 

-  $800M,  40  months:  “halfway”  through  integration  and  test 

-  1:1  IOC  after  $3B,  80  months 

•  CP-based  Incremental  Commitment  [number  of  competing  teams] 

-  $25M,  6  mo.  to  VCR  [4]:  may  beat  1 :2  with  agent  technology,  but  not  4:1 

-  $75M,  8  mo.  to  ACR  [3]:  agent  technology  may  do  1 :1 ;  some  risks 

-  $225M,  10  mo.  to  DCR  [2]:  validated  architecture,  high-risk  elements 

-  $675M,  18  mo.  to  IOC  [1]:  viable  1:1  capability 

-  1:1  IOC  after  $1 B,  42  months 
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Example  Risks  Involved  in  CP 

Based  on  TRW,  DARPA,  SAIC  experiences;  workshop 


*  Seductiveness  of  sunny-day  demos 

-  Lack  of  coverage  of  rainy-day  off-nominal  scenarios 

-  Lack  of  off-ramps  for  infeasible  outcomes 

*  Underemphasis  on  quality  factor  tradeoffs 

-  Scalability,  performance,  safety,  security,  adaptability 

*  Discontinuous  support  of  developers,  evaluators 

-  Loss  of  key  team  members 

-  Inadequate  evaluation  of  competitors 

*  Underestimation  of  productization  costs 

-  Brooks  factor  of  9  for  software 

-  May  be  higher  for  hardware 

*  Underemphasis  on  non-prototype  factors 
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Milestone  B  Focus  on  Technology  Maturity 
Misses  Many  OSD/AT&L  Systemic  Root  Causes 


1  Technical  process  (35  instances) 
-V&V,  integration,  modeling&sim. 

2  Management  process  (31) 

3  Acquisition  practices  (26) 

4  Requirements  process  (25) 

5  Competing  priorities  (23) 


6  Lack  of  appropriate  staff  (23) 

7  Ineffective  organization  (22) 

8  Ineffective  communication  (21) 

9  Program  realism  (21) 

10  Contract  structure  (20) 


•Some  of  these  are  root  causes  of  technology  immaturity 

•Can  address  these  via  evidence-based  Milestone  B  exit  criteria 
•Technology  Development  Strategy 
•Capability  Development  Document 

•Evidence  of  affordability,  KPP  satisfaction,  program  achievability 
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What  is  the  ICM? 


Risk-driven  framework  for  tailoring  system  life- 
cycle  processes 

Integrates  the  strengths  of  phased  and  risk-driven 
spiral  process  models 

Synthesizes  together  principles  critical  to 
successful  system  development 


-  Commitment  and  accountability  of  system  sponsors 

-  Success-critical  stakeholder  satisficing 


-  Incremental  growth  of  system  definition  and  stakeholder 
commitment 

-  Concurrent  engineering 


Principles 
trump 
diagrams ... 


-  Iterative  development  cycles 


-  Risk-based  activity  levels  and  evidence-based  milestones 


Principles  Used  by  60-80%  of  CrossTalk  Top-5  projects,  2002-2005 
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The  Incremental  Commitment  Life  Cycle  Process:  Overview 


Stage  I:  Definition 


Stage  II:  Development  and  Operations 


General/ 

DoD  Milestones 


*°v 
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<y 
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ICM 

Lifecycle  Phases 


Activities 


Concurrent  risk-and- 
opportunity-driven 
growth  of  system 
understanding  and 
definition 


Evaluation  of  evidence 
of  feasibility  to  proceed 


Initial  scoping 


Feasibility 

Evidence 


Concept 

definition 

Investment 

analysis 


System  life-cycle 
architecture  and 
ops  concept 

Build-to 

increment  plans 
and  specifications 

NDI,  outsource 
partner  selections 


Increment  1 
development 

Increment  2 
Foundations 
rebaseline 


Increment  1 
operations 

Increment  2 
development 

Increment  3 
Foundations 
rebaseline 


Synchronize,  stabilize  concurrency  via  FEDs 


I 


Risk  patterns 
determine  life  cycle 
process 
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ICM  HSI  Levels  of  Activity  for  Complex  Systems 
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Anchor  Point  Feasibility  Evidence  Description 


•  Evidence  provided  by  developer  and  validated  by 
independent  experts  that: 

If  the  system  is  built  to  the  specified  architecture,  it  will 

-  Satisfy  the  requirements:  capability,  interfaces,  level  of  service, 
and  evolution 

-  Support  the  operational  concept 

-  Be  buildable  within  the  budgets  and  schedules  in  the  plan 

-  Generate  a  viable  return  on  investment 

-  Generate  satisfactory  outcomes  for  all  of  the  success-critical 
stakeholders 

•  All  major  risks  resolved  or  covered  by  risk  management 
plans 

•  Serves  as  basis  for  stakeholders’  commitment  to  proceed 


Can  be  used  to  strengthen  current  schedule-  or  event-based  reviews 
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ICM  Nature  and  Origins 

*  Integrates  hardware,  software,  and  human  factors 
elements  of  systems  engineering 

-  Concurrent  exploration  of  needs  and  opportunities 

-  Concurrent  engineering  of  hardware,  software,  human  aspects 

-  Concurrency  stabilized  via  anchor  point  milestones 

*  Developed  in  response  to  DoD-related  issues 

-  Clarify  “spiral  development”  usage  in  DoD  Instruction  5000.2 

•  Initial  phased  version  (2005) 

-  Explain  Future  Combat  System  of  systems  spiral  usage  to  GAO 

•  Underlying  process  principles  (2006) 

-  Provide  framework  for  human-systems  integration 

•  National  Research  Council  report  (2007) 

*  Integrates  strengths  of  current  process  models 

-  But  not  their  weaknesses 

03/19/2008  ©USC-CSSE  15 


|c|s|s|e| 


University  of  Southern  California 

Center  for  Systems  and  Software  Engineering 


ICM  Integrates  Strengths  of  Current  Process  Models 

But  not  their  weaknesses 


*  V-Model:  Emphasis  on  early  verification  and  validation 

-  But  not  ease  of  sequential,  single-increment  interpretation 

*  Spiral  Model:  Risk-driven  activity  prioritization 

-  But  not  lack  of  well-defined  in-process  milestones 

*  RUP  and  MBASE:  Concurrent  engineering  stabilized  by 
anchor  point  milestones 

-  But  not  software  orientation 

*  Lean  Development:  Emphasis  on  value-adding  activities 

-  But  not  repeatable  manufacturing  orientation 

*  Agile  Methods:  Adaptability  to  unexpected  change 

-  But  not  software  orientation,  lack  of  scalability 
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Applying  ICM  Principles  and  Practices  to  CP 


•  When,  what,  and  how  much  to  prototype? 

-  Risk  management  principle:  buying  information  to 
reduce  risk 

•  Whom  to  involve  in  CP? 

-  Satisficing  principle:  all  success-critical  stakeholders 

•  How  to  sequence  CP? 

-  Incremental  growth,  iteration  principles 

•  How  to  plan  for  CP? 

-  Concurrent  engineering  principle:  more  parallel  effort 

•  What  is  needed  at  Milestone  B  besides 
prototypes? 

-  Risk  management  principle:  systemic  analysis  insights 
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When,  What,  and  How  Much  to  Prototype? 

-  Buying  information  to  reduce  risk 


•  When  and  what:  Expected  value  of  perfect 
information 

•  How  much  is  enough:  Simple  statistical 
decision  theory 


July  2008 


©USC-CSSE 


19 


|c|s|s|e| 


University  of  Southern  California 

Center  for  Systems  and  Software  Engineering 


When  and  What  to  Prototype:  Early  RPV  Example 


*  Bold  approach 

0.5  probability  of  success:  Value  VBS  =  $100M 
0.5  probability  of  failure:  Value  VBF  =  -  $20M 

*  Conservative  approach 

Value  VC  =  $20M 

*  Expected  value  with  no  information 

EVni  =  max(EVB,  EVC)  =  max(.5($100M)+.5(-$20M),  $20M) 

=  max($50M-$10M,$20M)  =  $40M 

*  Expected  value  with  perfect  information 

EVPI  =  0.5[max(VBs,VC)]  +  0.5[max(VBF,VC)] 

=  0.5  *  max($100M,$20M)  +  0.5  *  max(-$20M,$20M) 

=  0.5  *  $100M  +  0.5  *  $20M  =  $60M 

*  Expected  value  of  perfect  information 

EVPI  =  EVp,  -  EVni  =  $20M 

*  Can  spend  up  to  $20M  buying  information  to  reduce  risk 
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If  Risk  Exposure  is  Low,  CP  Has  Less  Value 

•  Risk  Exposure  RE  =  Prob(Loss)  *  Size(Loss) 

•  Value  of  CP  (EVPI)  would  be  very  small  if  the 
Bold  approach  is  less  risky 

-  Prob(Loss)  =  Prob  (VBF)  is  near  zero  rather  than  0.5 

-  Size(Loss)  =  VBF  is  near  $20M  rather  than  -$20M 
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How  Much  Prototyping  is  Enough? 

-  Value  of  imperfect  information 


•  Larger  CP  investments  reduce  the  probability  of 

-  False  Negatives  (FN):  prototype  fails,  but  approach  would  succeed 

-  False  Positives  (FP):  prototype  succeeds,  but  approach  would  fail 

•  Can  calculate  EV(Prototype)  from  previous  data  plus  P(FN),  P(FP) 


CP  Cost 

(dd)d 

P(FN) 

EV(CP) 

EV(lnfo) 

Net  EV(CP) 

0 

$40  M 

0 

0 

$2M 

0.3 

0.2 

$46M 

$6M 

$4M 

$5M 

0.2 

0.1 

$52M 

$12M 

$7M 

$10M 

0.15 

0.075 

$54M 

$14M 

$4M 

$15M 

0.1 

0.05 

$56M 

$16M 

$1M 

$22  M 

0.0 

0.0 

$60  M 

$20M 

-$2M 

•  Added  CP  decision  criterion 

-  The  prototype  can  cost-effectively  reduce  the  uncertainty 
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Summary:  CP  Pays  Off  When 

•  The  basic  CP  value  propositions  are  satisfied 

1.  There  is  significant  risk  exposure  in  making  the  wrong 
decision 

2.  The  prototype  can  cost-effectively  reduce  the  risk 
exposure 

*  There  are  net  positive  side  effects 

3.  The  CP  process  does  not  consume  too  much  calendar 
time 

4.  The  prototypes  have  added  value  for  teambuilding  or 
training 

5.  The  prototypes  can  be  used  as  part  of  the  product 
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Applying  ICM  Principles  and  Practices  to  CP 


•  When,  what,  and  how  much  to  prototype? 

-  Risk  management  principle:  buying  information  to 
reduce  risk 

•  Whom  to  involve  in  CP? 

-  Satisficing  principle:  all  success-critical  stakeholders 

•  How  to  sequence  CP? 

-  Incremental  growth,  iteration  principles 

•  How  to  plan  for  CP? 

-  Concurrent  engineering  principle:  more  parallel  effort 

•  What  is  needed  at  Milestone  B  besides 
prototypes? 

-  Risk  management  principle:  systemic  analysis  insights 
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Whom  to  Involve  in  CP? 

-  Satisficing  principle:  All  success-critical  stakeholders 


*  Success-critical:  high  risk  of  neglecting  their 
interests 


-  Acquirers 

-  Developers 

-  Users 

-  Testers 


-  Operators 

-  Maintainers 

-  Interoperators 

-  Others 


*  Risk-driven  level  of  involvement 

-  Interoperators:  initially  high-level;  increasing  detail 

*  Need  to  have  CRACK  stakeholder  participants 

-  Committed,  Representative,  Authorized,  Collaborative, 
Knowledgeable 
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How  to  Sequence  CP? 

-  Iterative  cycles;  incremental  commitment  principles 
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Actual  CP  Situation:  Need  to  Conserve  Momentum 


*  Need  time  to  evaluate  and  rebaseline 

•  Eliminated  competitors’  experience  lost 
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Keeping  Competitors  Productive  and  Supported 

During  Evaluations 

-  Concurrent  engineering  principle 


*  Provide  support  for  a  core  group  within  each 
competitor  organization 

-  Focused  on  supporting  evaluation  activities 

-  Avoiding  loss  of  tacit  knowledge  and  momentum 

*  Key  evaluation  support  activities  might  include 

-  Supporting  prototype  exercises 

-  Answering  questions  about  critical  success  factors 

*  Important  to  keep  evaluation  and  selection  period 
as  short  as  possible 

-  Through  extensive  preparation  activities  (see  next  chart) 


July  2008 


©USC-CSSE 


28 


|c|s|s|e| 


University  of  Southern  California 

Center  for  Systems  and  Software  Engineering 


Keeping  Acquirers  Productive  and 
Supported  During  Prototyping 

*  Adjusting  plans  based  on  new  information 

*  Preparing  evaluation  tools  and  testbeds 

-  Criteria,  scenarios,  experts,  stakeholders,  detailed 
procedures 

*  Possibly  assimilating  downselected  competitors 

-  IV&V  contracts  as  consolation  prizes 

*  Identifying,  involving  success-critical  stakeholders 

*  Reviewing  interim  progress 

*  Pursuing  complementary  acquisition  initiatives 

-  Operational  concept  definition,  life  cycle  planning,  external 
interface  negotiation,  mission  cost-effectiveness  analysis 
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Applying  ICM  Principles  and  Practices  to  CP 


•  When,  what,  and  how  much  to  prototype? 

-  Risk  management  principle:  buying  information  to 
reduce  risk 

•  Whom  to  involve  in  CP? 

-  Satisficing  principle:  all  success-critical  stakeholders 

•  How  to  sequence  CP? 

-  Incremental  growth,  iteration  principles 

•  How  to  plan  for  CP? 

-  Concurrent  engineering  principle:  more  parallel  effort 

•  What  is  needed  at  Milestone  B  besides 
prototypes? 

-  Risk  management  principle:  systemic  analysis  insights 
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Later  CP  Rounds  Need  Increasing 
Focus  on  Complementary  Practices 


-  By  all  success  critical  stakeholders 


*  Stakeholder  roles,  responsibilities,  authority, 
accountability 

*  Capability  priorities  and  sequencing  of 
development  increments 

*  Concurrent  engineering  of  requirements, 
architecture,  feasibility  evidence 

*  Early  preparation  of  development  infrastructure 
(i.e.,  key  parts  of  the  architecture) 

*  Acquisition  planning,  contracting,  management, 
staffing,  test  and  evaluation 


July  2008 


©USC-CSSE 


31 


III|E| 


University  of  Southern  California 

Center  for  Systems  and  Software  Engineering 


When  to  Stop  CP 

-  Commitment  and  accountability  principle:  Off-ramps 


*  Inadequate  technology  base 

-  Lack  of  evidence  of  scalability,  security,  accuracy, 
robustness,  airworthiness,  useful  lifetime, ... 

-  Better  to  pursue  as  research,  exploratory  development 

*  Better  alternative  solutions  emerge 

-  Commercial,  other  government 

*  Key  success-critical  stakeholders  decommit 

-  Infrastructure  providers,  strategic  partners,  changed 
leadership 


Important  to  emphasize  possibility  of  off-ramps.... 
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Acquiring  Organization’s  ICM-Based  CP  Plan 


•  Addresses  issues  discussed  above 

-  Risk-driven  prototyping  rounds,  concurrent  definition  and 
development,  continuity  of  support,  stakeholder  involvement, 
off-ramps 

•  Organized  around  key  management  questions 

-  Objectives  (why?):  concept  feasibility,  best  system  solution 

-  Milestones  and  Schedules  (what?  when?):  Number  and  timing 
of  competitive  rounds;  entry  and  exit  criteria,  including  off¬ 
ramps 

-  Responsibilities  (who?  where?):  Success-critical  stakeholder 
roles  and  responsibilities  for  activities  and  artifacts 

-  Approach  (how?):  Management  approach  or  evaluation 
guidelines,  technical  approach  or  evaluation  methods,  facilities, 
tools,  and  concurrent  engineering 

-  Resources  (how  much?):  Necessary  resources  for  acquirers, 
competitors,  evaluators,  other  stakeholders  across  full  range  of 
prototyping  and  evaluation  rounds 

-  Assumptions  (whereas?):  Conditions  for  exercise  of  off-ramps, 
rebaselining  of  priorities  and  criteria 

•  Provides  a  stable  framework  for  pursuing  CP 
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Outline 


•  Motivation  and  Context 

•  Nature  of  the  ICM 

•  Applying  ICM  Principles  to  CP 

•  Conclusions,  References,  Acronyms 

-  Copy  of  Young  Memo 
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CP  Conclusions 


•  CP  most  effective  in  reducing  technical  risk 

-  If  project  is  low-risk,  may  not  need  CP 

•  May  be  worth  it  for  teambuilding 

•  Other  significant  risks  need  resolution  by  Milestone  B 

-  Systemic  Analysis  DataBase  (SADB)  sources:  management, 
acquisition,  requirements,  staffing,  organizing,  contracting 

•  CP  requires  significant,  continuing  preparation 

-  Prototypes  are  just  tip  of  iceberg 

-  Need  evaluation  criteria,  tools,  testbeds,  scenarios,  staffing, 
procedures 

•  Need  to  sustain  CP  momentum  across  evaluation  breaks 

-  Useful  competitor  tasks  to  do;  need  funding  support 

•  ICM  provides  effective  framework  for  CP  plan,  execution 

-  CP  value  propositions,  milestone  criteria,  guiding  principles 

•  CP  will  involve  changes  in  cultures  and  institutions 

-  Need  continuous  corporate  assessment  and  improvement  of 
CP-related  principles,  processes,  and  practices 
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List  of  Acronyms 


CD 

Concept  Development 

ICM 

Incremental  Commitment  Model 

CP 

Competitive  Prototyping 

KPP 

Key  Performance  Parameter 

DCR 

Development  Commitment 

Review 

MBASE 

Model-Based  Architecting  and 
Software  Engineering 

DoD 

Department  of  Defense 

OCR 

Operations  Commitment 

ECR 

Exploration  Commitment 

Review 

Review 

P(FN) 

Probability  of  False  Negatives 

EV 

Expected  Value 

P(FP) 

Probability  of  False  Positives 

EVNI 

Expected  Value,  No  Information 

RE 

Risk  Exposure 

EVPI 

Expected  Value,  Perfect 

RUP 

Rational  Unified  Process 

Information 

V&V 

Verification  and  Validation 

FCR 

Foundations  Commitment 

Review 

VB 

VBS 

Value  of  Bold  approach 

VB  for  success 

FED 

Feasibility  Evidence  Description 

VBF 

VB  for  failure 

GAO 

Government  Accounting  Office 

VC 

VCR 

Value  of  Conservative  approach 
Valuation  Commitment  Review 
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Competitive  Prototyping  Policy:  John  Young  Memo 


THE  UNDER  SECRETARY  OF  DEFENSE 

3010  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3010 

19  SEP  2007 

ACQUISITION, 

TECHNOLOGY 
AND  LOGISTICS 

MEMORANDUM  FOR  SECRETARIES  OF  THE  MILITARY  DEPARTMENTS 
CHAIRMAN  OF  THE  JOINT  CHIEFS  OF  STAFF 
COMMANDER,  U.S.  SPECIAL  OPERATIONS  COMMAND 
DIRECTORS  OF  THE  DEFENSE  AGENCIES 

SUBJECT:  Prototyping  and  Competition 

Many  troubled  programs  share  common  traits  -  the  programs  were  initiated  with 
inadequate  technology  maturity  and  an  elementary  understanding  of  the  critical  program 
development  path.  Specifically,  program  decisions  were  based  largely  on  paper 
proposals  that  provided  inadequate  knowledge  of  technical  risk  and  a  weak  foundation 
for  estimating  development  and  procurement  cost.  The  Department  must  rectify  these 
situations. 

Lessons  of  the  past,  and  the  recommendations  of  multiple  reviews,  including  the 
Packard  Commission  report,  emphasize  the  need  for,  and  benefits  of,  quality  prototyping. 
The  Department  needs  to  discover  issues  before  the  costly  System  Design  and 
Development  (SDD)  phase.  During  SDD,  large  teams  should  be  producing  detailed 
manufacturing  designs  -  not  solving  myriad  technical  issues.  Government  and  industry 
teams  must  work  together  to  demonstrate  the  key  knowledge  elements  that  can  inform 
future  development  and  budget  decisions. 

To  implement  this  approach,  the  Military  Services  and  Defense  Agencies  will 
formulate  all  pending  and  future  programs  with  acquisition  strategies  and  funding  that 
provide  for  two  or  more  competing  teams  producing  prototypes  through  Milestone  (MS) 
B.  Competing  teams  producing  prototypes  of  key  system  elements  will  reduce  technical 
risk,  validate  designs,  validate  cost  estimates,  evaluate  manufacturing  processes,  and 
refine  requirements.  In  total,  this  approach  will  also  reduce  time  to  fielding. 

Beyond  these  key  merits,  program  strategies  defined  with  multiple,  competing 
prototypes  provide  a  number  of  secondary  benefits.  First,  these  efforts  exercise  and 
develop  government  and  industry  management  teams.  Second,  the  prototyping  efforts 
provide  an  opportunity  to  develop  and  enhance  system  engineering  skills.  Third,  the 
programs  provide  a  method  to  exercise  and  retain  certain  critical  core  engineering  skills 
in  the  government  and  our  industrial  base.  Fourth,  prototype  efforts  can  attract  a  new 
generation  of  young  scientists  and  engineers  to  apply  their  technical  talents  to  the  needs 
of  our  Nation’s  Warfighters.  Finally,  these  prototype  efforts  can  inspire  the  imagination 
and  creativity  of  a  new  generation  of  young  students,  encouraging  them  to  pursue 
technical  educations  and  careers. 


Based  on  these  considerations,  all  acquisition  strategies  requiring  lJSD(.A!'&L) 
approval  must  be  formulated  to  include  competitive,  technically  mamse  prototyping 
through  MS  B.  The  Component  Acquisitions  Executives  will  review  all  existing 
programs  and  all  programs  in  the  initial  stages  of  development  for  the  potential  to  adopt 
this  acquisition  strategy.  It  is  the  policy  of  the  Depart  mem.  of  Defense  that  this 
acquisition  strategy  should  be  extended  to  all  appropriate  program*  below  AC  AT  [. 


Under  Secretaries  Of  Defense 
Component  Acquisition  Executives 
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Synchronizing  M&S  Plans  Across 

Navy  Acquisition 


Or: 

Prior  Proper  Prudent  Planning  Prevents 
Piss  Poor  Performance 


21  October  2008 


The  insights  presented  here  are  those  of  the  authors  and  do  not  reflect  any  official  views. 
We  would  like  to  thank  NAVMSMO,  NRL,  and  NMSO  for  their  years  of  support. 


Ivar  Oswalt  and  Robert  Tyler 
VisiTech 

Suite  400,  500  Montgomery  Street 
Alexandria,  Virginia  22314 

Oswalt@VisiTech.com  -  Tvler@VisiTech.com 


Outline 


•  Why  M&S  Planning 

•  Synchronize  to  Requirements 

•  Synchronize  to  Other  Plans 

•  Synchronize  to  Future  Activities 

•  Snapshot  of  Current  Navy  Acquisition  M&S  Plans 

•  Conclusions  and  Recommendations 
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M&S  Investments 


VisiTech 


Large  Expenditures  in  M&S! 

Yet  the  Cost  of  Planning  is  Quite  Low* 
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M&S  Planning  Can... 


*  Identify  cross-cutting  requirements  and  potential  synergies 

*  Associate  funding  expenditures  and  capability  delivery 

*  Facilitate  common  technical  infrastructures 

*  Establish  relationships  between  key  personnel 

*  Help  coordinate  individual  efforts 
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Synchronize  to  Requirements 


Relevance  to  Community 


•  Leaders  /  Sponsors 


•  Planners  /  Managers 

•  Developers  /  Implementers 

•  Operators  /  Users 


That  is: 

Leave 

Any 

One 

Out 

at 

Your 

Peril 
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Synchronize  to  Requirements 


Empower  and  Involve 


•  The  Willing 


•  The  Impacted 


•  The  Needed 


•  The  Required 
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Synchronize  to  Requirements 


Guide  and  Iterate 


•  Spiral  Development  -  With  POA&M 


•  Include  All  Community  Members 


•  Start  General  and  Mature  Specificity 


•  Stay  in  ‘Swim  Lane’  of  Plan  Type 
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Synchronize  to  Other  Plans 


Interconnect  to  Other  Documents 


Baseline  Assessment 

Descriptive 

Application  Areas  (and  enablers) 
Programs  and  Funding 
Trends  and  Issues 


Application  and  Enabler  Road  Maps 

Proactive 

Area  visions  and  goals 
Implementation,  Objectives,  Funding,  Metrics 
Processes,  Resources,  Timing 
Organizational  Missions 


Master  Plan 

Prescriptive 

Navy-Wide  Vision  and  Goal 
Application  Areas  and  Enablers 
(associated  visions  and  goals) 
Strategies  and  Sequencing 
NAVMSMO’s  Mission 


Investment  Strategy 

Fiscal 

Simulation  and  Supporting  Efforts 
Analysis  and  Navy-Wide  Rationalization  and  POMing 
Dollars,  Personnel,  and  Facilities  with  Potential  for  Synergy 


Slide  6- 18  May  2004 
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Synchronize  to  Other  Plans 


Account  for  External  Activities 

External  Depends... 

•  Coalition,  Joint,  DoD,  Other  Service,  ... 

•  DHS,  DoS,  ... 

•  Congressional,  ... 

•  Considering  Each  /  All  Can  Improve  the  Plan 


VisiTech 
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Synchronize  to  Other  Plans 


Define  Terms  and  Timing! 


Master 

Plan 

Vision  =  Utility  of  simulation  to  the  Navy  Enterprise 

Goals  =  Application  and  enabler  sub-components  of  Navy-wide  vision  (long-term) 

“5-25  yrs” 

Strategy  =  Overall  actions  taken  to  reach  vision  and  goals 

Business 

Plan 

Vision  =  Contribution  of  simulation  to  application  area  or  enabler  (goals  from  above) 
Goals  =  More  specific  application  area  or  enabler  components  (mid-term) 

“3-15  yrs” 

Means  =  What  needs  to  be  done  and  process  improvements 

Mission  =  Relevant  organizational  roles  and  responsibilities 

Road 

Map 

Goals  =  Application  of  M&S  to  meet  systemic  goals 

Objectives  =  Activities  and  tasks  required  to  achieve  goals  (mid  /  near-term) 

Execution  Approach  =  Means  to  accomplish  goals  and  objectives  (how,  who,  where) 

“1-10  yrs” 

Sequencing,  Timing,  Resources  =  Order,  duration  (when),  and  investments  needed 
Metrics  =  That  reflect  contribution  /  value  and  degree  to  which  objectives  have  been  met 

Implementation 

Guide 

Execution  Approach  =  Specific  steps  /  actions  required  (near  /  now-term) 

Context  =  Application  of  individual  standards,  codes  of  best  practice,  and  similar 

Product  =  A  POA&M  of  capabilities  that  will  be  delivered  over  time 

“ 1-3  yrs” 

and  Strategic  Plan ,  Investment  Strategy,  Program  Plan, ... 

VisiTech 
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Synchronize  to  Future  Activities 


Implementing  Processes 

•  M&S  Specific  -  Technical 

-  Visualization,  Data,  Time  Management 

-  Languages  (JAVA),  Availability  (SOA),  etc. 

-  Hardware  /  Distribution  Alignment 

•  M&S  Context  -  Requirements 

-  Information  Technology,  Soft/Hardware,  etc. 

-  R&D  /  S&T  /  ACTDs 

-  Commercial  Developments 

•  M&S  Relevant  -  Users 

-  Involve  End  Users  Early  and  Often 

-  Understand  and  Reflect  the  Problem  Context  with  the  M&S  Use 
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Synchronize  to  Future  Activities 


Incorporation  of  Data 


•  Good 

•  And  Evolve,  to... 

•  Good  Enough 


•  “The  perfect  is  the  enemy  of  the  good  enough” 
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Synchronize  to  Future  Activities 


Integration  of  Leadership 


•  Involve  Leader  (s)  -  AMAP 


•  Develop  Broad  “Top  Cover” 


•  Iterate  /  Promulgate  Ideas,  Plans,  Policies 


•  Implement  (Enforce)  Directives 


VisiTech 


(as  much  as  possible!) 
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Synchronizing  M&S  Plans  Across  Navy  Acquisition 

Snapshot  of  Today 

•  The  US  Navy  M&S  Acquisition  Community  has 

-  Developed  an  M&S  Business  Plan  Structure 

-  Using  it  as  a  foundation  for  an  ASN(RDA)  M&S  Road  Map 

-  RM  includes  Leadership,  Infrastructure,  and  Similar 

-  RDA  interacts  with  all  Navy  M&S  Communities 

-  “Lead  by  Example  While  Gathering  Steam” 
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Conclusion  -  Planning  Can  Establish 


•  “Shared  vision  /  understanding  of  objectives 

•  Commitment  of  the  organization  and  its  people 

•  Ability  to  partition  complexity  into  actionable  parts 

•  Use  of  intermediate  steps 

•  Application  of  proven  methods  and  standards”* 


VisiTech 


*  Success  Factors,  SOS  Engineering  Conference,  25  July  2006,  Mr.  Carl  Siel,  ASN(RDA)  CHENG 
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Recommendation 


•  Plan! 


•  It's  well  worth  the  investment! 
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Plans  Promote...  (I) 


1 .  Conversion  of  the  vision,  goals,  and  strategy  found  in  the  Master  Plan  into 
specific  (executable)  actions  and  objectives 

2.  Better  meeting  of  requirements  through  articulation,  projection,  and 
understanding  of  needs  and  capabilities  available  to  address  them 

3.  System  life-cycle  cost  reduction  by  efficiently  meeting  requirements  and 
through  enabler  alignment,  synergy,  and  integration 

4.  Identification  of  system,  decision,  and  process  prerequisites,  precedence, 
dependence,  and  sequencing 

5.  Establishment  of  technology  insertion  and  modernization  points  and  ways  to 
leverage  other  Service,  Joint,  Government,  and  private  enterprise  initiatives 

6.  Definition  of  current  and  needed  funding  levels,  programmatics,  and 
relevant  performance  metrics 

7.  Capabilities  development,  acquisition,  and  deployment  priorities  and 
approaches 

8.  Identification  of  organizational  roles  and  responsibilities  and  proposed 
changes  and  enhancements 


VisiTech  -  to  include  warfighter  impact,  opportunity  costs,  and  similar  measures  of  merit 
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Road  Maps  Promote...  (II) 


9.  Statement  of  definitions,  informing  interested  communities,  and  consolidation 
of  relevant  information,  resources,  and  references 

10.  Base-lining  of  current  systems  and  developing  consensus  on  requirements 

1 1 .  Plans  to  be  formulated  to  meet  current  requirements  and  proactive  approaches 
to  be  constructed  to  address  long  term  needs 

12.  Development  and  agreement  on  process  descriptions  of  needed  and  optional 
actions,  decisions,  information  gathering  and  submission  points,  and  roles  and 
responsibilities  of  organizations  and  individuals 

13.  Effective  orchestration  of  experiments,  demonstrations  (ACDs  and  ACTDs), 
systems  developments  and  deployment,  and  organizational  changes 

14.  System  convergence,  integration,  and  consolidation  approaches  that  may  be 
required 

1 5.  Characterization  of  challenges  and  approaches  to  meet  them 

16.  Matching  and  aligning  of  future  required  capabilities,  emerging  software  and 
hardware  technologies,  developing  standards,  and  maturing  design, 
development,  and  manufacturing  methods 
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Process  Enrichment^  Boot  Camp 

An  intensive  introduction  to  a  generic,  enterprise-wide,  strategic 
communication  and  continuous  improvement  methodology 


Presented  to  The  National  Defense  Industrial  Association 

October  23,  2008 

Victor  Elias 

High  Performance  Technologies,  Inc. 


Process  Enrichment3*  Boot  Camp 

Briefing  Outline 


Strategic  Communication 
Defined 

Themes  of  Performance 
A  Thematic  Strategic  Management  System 
Identifying  Measurements  of  Strategic  Performance 

The  Art  of  Process  EnrichmentSM  in  Competitive  Warfare 

Quality  Excellence:  A  New  Definition 

Case  Studies:  Assessing  Competitive  Position 

Case  Study:  Assessing  Market  Value 

Beyond  Excellence:  The  Quest  for  Process  EnrichmentSM 

Reversing  De-motivating  Conditions 
Supporting  an  Innovative  Culture 
Improving  Your  Customer's  Products  &  Services 
Transformation  to  Serve  Emergent  Market  Needs 
A  Systems  View  of  Continuous  Improvement 

Conclusion  /  Q  &  A 


An  intensive  introduction  to  a  generic,  enterprise-wide,  strategic  communication  and  continuous  improvement  methodology 


Process  Enrichment*  Boot  Camp 

Strategic  Communication:  Defined 


"Focused  United  States  Government  efforts 
to  understand  and  engage  key  audiences 
to  create,  strengthen,  or  preserve  conditions  favorable 
for  the  advancement  of  United  States  Government 

interests,  policies,  and  objectives  through  the  use  of 
coordinated  programs,  plans,  themes,  messages,  and  products 
synchronized  with  the  actions  of  all  instruments  of  national  power." 

-  Joint  Chiefs  of  Staff 

JP  5-0 


(For  "United  States  Government"  and  "national"  read  "Enterprise") 


An  intensive  introduction  to  a  generic,  enterprise-wide,  strategic  communication  and  continuous  improvement  methodology 


Process  Enrichment3*1  Boot  Camp 

Strategic  Communication:  The  Message 


onai:  rigni  on  me  beacn 

June  4,  1940,  House  of  Commons 


We  shall  prove  ourselves  once  more  able  to  defend  our  Island  home... 

...  We  shall  go  on  to  the  end,  we  shall  fight  in  France,  we  shall  fight  on  the  seas  and 
oceans,  we  shall  fight  with  growing  confidence  and  growing  strength  in  the  air,  we  shall 
defend  our  Island,  whatever  the  cost  may  be,  we  shall  fight  on  the  beaches,  we  shall  fight 
on  the  landing  grounds,  we  shall  fight  in  the  fields  and  in  the  streets,  we  shall  fight  in  the 
hills;  we  shall  never  surrender,  and  if,  which  I  do  not  for  a  moment  believe,  this  Island  or  a 
large  part  of  it  were  subjugated  and  starving,  then  our  Empire  beyond  the  seas,  armed  and 
guarded  by  the  British  Fleet,  would  carry  on  the  struggle,  until,  in  God's  good  time,  the  New 
World,  with  all  its  power  and  might,  steps  forth  to  the  rescue  and  the  liberation  of  the  old.” 


Listen  To  Speech 


An  intensive  introduction  to  a  generic,  enterprise-wide,  strategic  communication  and  continuous  improvement  methodology 
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Strategic  Communication:  The  Structure 


Conceptually, 

Market  Incentives  and  Expectations  regarding  them  produce  Motives  expressed  as 

Intentions  that  are  carried  out  through  Behaviors  using  Resources. 


Expectations: 

Situational 

Awareness: 

Market 

Analysis 


"We  shall  go  on  to  the  end" 
"Liberation" 


m 


Resources 

Means 


Preserve  Democratic 
Government 


"Victory" 

Preserve  Independence 


"defend  our  island  home" 


"our  Empire"  "we  shall  fighf 


V/ 
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Strategic  Communication:  Operationalizing  The  Message 


Process  Enrichments'  concepts  of 

and 


Themes  of  Porlfotlftance 


serve  as  a  common  language 
that  enables  a  systems  engineering  process 


Expectati  ons: 

Situational 

Awareness; 

Market 

Analysis 


lesources 

Means 


Behaviors 

Ways 

Strategy 


Intentions 

Ends 

Objectives 


Motives 

Political  Purpose 

Vision  &  Mission 


Performance 


to  efficiently  enact  strategic  communication 
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Strategic  Communication:  The  Enabling  Concepts 


4 


World  Class 
Perform  since 

Process 

Enrichment 


Optimization 


Sustainment 


Efficacy 


Acceptance 


Themes 
□f 

Perfo  rmance 


Anything 

considered  in  terms  of  its  performance 


Products! 


ts  a 


c  An  Organization 
A  Project 
->  A  Service 
j  A  Product 
j  A  Process 


Themes  of  Performance 

are  a  set  of  6 
hierarchical,  independent, 

"Themes1" 
that  comprehensively  describe 
the  performance  qualities 
of  any 


(^Themes:  implicit,  recurring,  and  coalescent  central  concepts,  principles,  qualities  and/or  ideas) 
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Strategic  Communication:  Themes  of  Performance 


If  the  Performance  Unit  is  Prod  yet  -  for  example:  Projectile  XYZ 


Acceptance 


Efficacy 


Sustainment 


Optimization 


Projectile  XYZ 


Enrichment^ 

World  Class 
Performance 


Does  Projectile  XYZ  fit  in 
the  launcher? 

Will  Projectile  XYZ  destroy 
the  target? 

How  many  Projectile  XYZ's 
can  be  made  per  month? 

Can  I  make  Projectile  XYZ 
more  dangerous  to  the 
target? 

Can  I  transport  Projectile 
c  XYZ  without  it  exploding? 

Can  other  allied  troops  use 
Projectile  XYZ? 
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Strategic  Communication:  Themes  of  Performance 


If  the  Performance  Unit  is  A  Procedure  -  for  example:  Procedure  XYZ 


Acceptance 


Efficacy 


Sustainment 


Optimization 


Process 

Enrichment5^ 

World  Class 
Performance 


Can  Procedure  XYZ  be 
performed  without  "dropping 
the  ball"  with  respect  to 
stakeholder  interests? 

Does  Procedure  XYZ  make  a 
useful  contribution  to  how  we 
do  things? 

Can  repetitive  Procedure  XYZ 
be  repeated? 

Is  there  One  Best  Way  to 
perform  Procedure  XYZ? 

Are  people  working  against 
each  other? 

Is  Procedure  XYZ  performed 
the  same  way  everywhere  it's 
performed? 
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Strategic  Communication:  Themes  of  Performance 


If  the  Performance  Unit  is  Ap  Organization  -  for  example,  a  Division  responsible 
for  Policies,  Procedures,  Templates,  Tools  &  Training  ("guidance  products") 


Accept  &  nee 


Efficacy 


stammer 


Optimization 


EnriehmentSM 

World  Class 
Performance 


Are  all  necessary  guidance 
products  ready? 

Are  people  using  the 
guidance  products? 


Is  the  organi; 
^manageable? 


=Are  all  practices  Best 
Practices? 

Is  the  organization  using 
the  best  ideas  employees 
put  forward? 

Does  the  organization  have 
marketable  competitive 
advantages  in 
performance7 
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Strategic  Communication:  Themes  of  Performance 


Themes  of  Performance 

Performance  Unit :  The  Message  - 
Expression  of  Strategic  Motives  &  Intentions 

,  Acc^p|ar»ce 

Focuses  on  adequacy  of  Performance  Unit  content 

Efficacy 

Focuses  on  the  capacity  of  the  Performance  Unit  to 

produce  the  desired  results 

We  shall  go  on  to  the  end.  We  shall  fight  in  France  and 
on  the  seas  and  oceans; 

we  shall  fight  with  growing  confidence  and  growing 
strength  in  the  air.  We  shall  defend  our  island  whatever 
the  cost  may  be;  we  shall  fight  on  beaches,  landing 
grounds,  in  fields,  in  streets  and  on  the  hills. 

Sustainment 

Focuses  on  maintaining  support  of  the  ongoing 
operational  capability  covered  by  the  Performance  Unit 

We  shall  never  surrender 

Optimization 

Focuses  on  engineering  the  Performance  Unit  to 
achieve  its  design-best  implementation 

Process  Enrichment 
Focuses  on  optimizing  Performance  Unit  behavioral 
impact  (employee/customer  motivation)  and  usage 
process  implementation  (ease  of  performance. 

satisfaction,  etc.) 

and  if,  which  1  do  not  for  the  moment  believe,  this  island 
or  a  large  part  of  it  were  subjugated  and  starving, 

then  our  empire  beyond  the  seas,  armed  and  guarded  by 
the  British  Fleet,  will  carry  on  the  struggle  until  in  God's 
good  time  the  New  World  with  all  its  power  and  might, 
sets  forth 

World  Class  Performance 
Focuses  on  interchangeability  and  aggregate  benefit  of 

the  Performance  Unit 

to  the  liberation  and  rescue  of  the  Old. 
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Preserve  Parliamentary  Democracy 


Sustainment 

We  shall  never 
surrender 


Optimization 

and  if,  which  I  do 
not  for  the  moment 
believe,  this  island 
or  a  large  part  of  it 
were. subjugated 
and  starving, 


Acceptance 

We  shall  go  on  to 
the  end.  We  shall 
fight  in  France  and 
on  the  seas  and 
oceans; 


Efficacy 

we  shall  fight  with 
growing 
confidence  and 
growing  strength  in 
the  air.  We  shall 
defend  our  island 
whatever  the  cost 
may  be;  we  shall 
fight  on  beaches, 
landing  grounds,  in 
fields,  in  streets 
and  on  the  hills. 


World  Class 
Performance 

to  the  liberation 
and  rescue  of  the 
Old. 


Process 

Enrichment*'* 

then  our  empire 
beyond  the  seas, 
armed  and 
guarded  by  the 
British  Fleet,  will 
carry  on  the 
struggle  until  in 
God's  good  time 
the  New  World  with 
all  its  power  and 
might,  sets  forth 


Incentive 


Vision 


Mission: 


Themes 


World  Class 
Performance 

Focuses  on 
interchangeability  and 
aggregate  benefit  of 
the  Performance  Unit 


Process 

Enrichments 

Focuses  on  optimizing 
Performance  Unit 
behavioral  impact  and 
usage  process 
implementation 


territory;  restore 

overthrown 

governments 


Win  all  battle: 


Never  give  in 


Optimization 

Focuses  on 
engineering  the 
Performance  Unit  to 
achieve  its  design-b 
implementation 


Communicate 
the  rationale  for  not 
surrendering; 
rationale  for  new 
resistance 


Communication  to 
Motivate  resistance 


Compose  fon 
appropriately  for 


Sustainment 

Focuses  on 
maintaining  support  of 
the  ongoing  operational 
capability  covered  by 
the  Performance  Unit 


es  commit 


■ .  Use  Strategic 
Communication  to 
promote  mutual 
aspirations 


ork  with  allie: 
rds  common 


Integrate  lesson: 
learned  into  plans  £ 


Efficacy 

Focuses  on  the  capacity 
of  the  Performance  Unit 
to  produce  the  desired 
results 


Gain  unconditional 
;urrender  of  all 
memy  combatant 


Dominate  the 
enemy's  message; 
Communicate  about 


::  Add/improve 
weaponry  &  soldiers  i 
leeded  competencies 


■  Allocate  resource 
as  needed;  find  new 


©  liberation 


‘ .  No  negotiation 
with  enemy 
combatant  nations 


Acceptance 

Focuses  on  adequacy 
of  Performance  Unit 
content 


Process  Enrichment^  Boot  Camp 

Strategic  Communication 


“Strategic  Communication  is  the  active  ingredient 

in  a  systems  engineering  process 

that  integrates 

the  essential  innovative  and  creative  direction 

of  the  enterprise’  guiding  motives 
and  creates  enduring  enterprise  performance  quality  excellence.” 

—  Victor  Elias 
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Identifying  Measurements  of  Strategic  Performance 


Theme:  An  implicit,  recurring,  and  coalescent 
central  concept,  principle,  quality  and/or  idea 
about  how  a  performance  Unit  can  perform. 

Focus  Area:  An  aspect  of  performance  that's 
implied  by  the  Theme 


[  Themes  1 


Acceptance 


Efficacy 


Sustainment 


Optimization 


Process 

Enrichment 


World  Class 
Performance 


Acceptance 


Content  Coverage 


Standards 

Compliance 


BusinessRules 

Compliance 


PeerReview 


Optimization 


Quality  Assurance 


Innovation 


of  Performance  Unit  content 
fthe  Performance  Unit  to  pro 
ng  support  of  the  ongoing  ope 
ng  the  Performance  Unit  to  a 
optimizing  Performance  Unit 
(ease  of  performance,  satisfi 
;  on  interchangeability  and  ag 


Effi  cacy 


Content  Suitability 


Process 
Enri  chment 


Motivational 

Structure 


Scenario  Utility 


Sustai  nment 


Operational 

Capability 


Communication 


Financial  Control 


World  Class 
Performance 


Resource 

Interchange 
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Identifying  Measurements  of  Strategic  Performance 


Behaviors 

Ways 

Strategy 


Content  Coverage 


Content  Coverage 


Measurements 


Focus  Area 


Focus  Area 


Focus  Area 


Business  Rules 


Focus  Area 


Focus  Area 


Indicator  Group:  A  group  of  indicators 
implied  by  the  Focus  Area 


Intentions 

Ends 

Objectives 


Motives 

Political  Purpose 

Vision  &  Mission 


Focused  Question:  A  context-relevant 
question  implied  by  the  Vision  &  Mission, 
Objective,  Strategy,  Theme,  Focus  Area,  and 
Indicator  Group  that  can  be  answered  by  a 
measurement 


Acceptance  ■■ 

Content  Coverage 

Standards 

Compliance 

Business  Rules 
Compliance 

Peer  Review 


Themes 


Indicator  Groups 


Acceptance 


Standards  Compliance  | 

Standards  Compliance 


Efficacy 


Sustainment  — 


Indicator  Groups 


Optimization 


Process 

Enrichment 


Indicator  Groups 


Peer  Review  HHHI 

Quality 

Feedback  Loop  Efficiency 


World  Class 
Performance 


Note:  Some  Focus  Areas  have  only  O  Indicator  Group,  other  Focus  Areas  have  up  to  ©. 
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Process  Enrichment31"1  Boot  Camp 

The  Art  of  Process  Enrichment  in  Competitive  Warfare 


World  Ctal 

Perfonmn 


Sustainment 


decisive 

Points 


The  battlefront  is  a  line  through 
the  Decisive  Points  in  each 
Theme  of  Performance.  The 
battlefront  holds  the  shape  of 
the  strategy  and  it  establishes 
the  competitive  position 


Performance  Perfection 

Performance  Units  scoring  6  in  all  6  Themes  of 
Performance  are  considered  perfect 


Decisive  Points 


A  Decisive  Point  is  a  critical 
part  of  the  battlefront  where 
an  advance  or  loss  in 
performance  quality  could  be 
decisive  on  the  outcome  of 
battle 


Efficacy 


2  1  * -  Scoring 

As  shown  by  the  position  of  each  Decisive 
Point,  each  Theme  of  Performance  has  been 
rated  on  a  scale  from  zero  (poorest)  on  the 
right  &  six  (best)  on  the  left  of  the  pyramid 


Process 

Enrichment5** 


Optimization 


Themes 
of 

Performance 


Acceptance 


Battlefront 


Battlefront 


is 
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Quality  Excellence:  Necessary  Improvements 


Wofld  Glass 
Performance 


Eralthmairt8*1 

optimization 

Sustainment 


"It  is  no  use  saying ,  'We  are 
doing  our  best . '  You  have 
got  to  succeed  in  doing  what 
is  necessaty . " 

--  Winston  Churchill 


N  M 


Efficacy 


Acceptance  A 


Quality  Excellence  as  shown,  is  the  highest  level  of  performance  quality  -  represented 
by  the  green  curve  through  the  decisive  points  in  each  Theme  of  Performance  -  that  a 
majority  of  customers  in  the  target  market  are  ready,  willing,  and  able  to  pay  for. 

The  gap  in  performance  quality  rating  from  a  ,  as  shown  in  Blue,  to 

the  quality  excellence  rating,  should  serve  as  a  clear  mandate  to  plan  and  implement 
necessary  improvements,  until  the  green  rating  is  achieved. 
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Quality  Excellence:  The  Risk  of  Poor  Performance 


<SJ  a 

Wotld  Class 
Performance 


xm  A 


process 

Element*" 

*)  ptimization  A 


Sustainment  A 


Efficacy  A 


Acceptance  A 


/  f 


S  A 
/  K 


The  deficit  in  performance  quality  in  the  zone  between  "  a  "  and  "  0  represents  a 
competitive  because  it  serves  as  an  incentive  to  new  entrants  or  other  competitors 
who  may  find  this  gap  in  performance  (failure  to  make  necessary  improvements)  a 
challenge  that  they  can  fulfill. 

This  is 
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Quality  Excellence:  The  Consequences  of  Poor  Performance 


Product:  has  50%  of  the  market  share 

and  new.  Competing  Product  or  Service  (C)  has  the 
other  50%. 

has  lost  50%  of  its  market  share  to  (C)  as  a 
consequence  of  the  realized 

Poor  performance  was  an  incentive  for 
(C)  to  enter  the  market. 

The  cost  of  quality  for  ,  so  far,  has  been  50%  of 
their  market  share. 

If  (C)  improves  it's  Acceptance  Theme  performance 
so  it's  better  than  ,  (C)  should  expect  to  secure 
100%  of  the  market  share  -  putting  out  of 
business.* 

For  (C),  if  the  value  of  the  additional  50%  of  the 
market  share  is  greater  than  the  investment  to 
exceed  's  performance  -  they  should  do  it  -  and 
they  should  plan  to  continue  improving  up  to  (Q). 

^Assuming  equivalent  competitive  circumstances  for  decision  factors  other  than  performance  (i.e.  convenience, 
loyalty,  selling  to  relatives,  etc.) 


fm  antv 


^ocess 

tr.ichmef 


3timizatior®i  A 


&  Sustainment A 


Efficacy 


Acceptance  A 
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Quality  Excellence:  A  New  Definition 


"Quality  Excellence 
is  the  absence  of 
the  risk  of  poor  performance 
in  each  Theme  of  Performance 

-  Victor  Elias 
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Case  Study 


The  War  To  End  All  Wars  (WWI) 
Assessing  Competitive  Position 


" There  was  no  other  point 
on  any  of  the  war  fronts , 
extending  over  hundreds 
of  miles,  where  an  equal 
advance  could  achieve 
the  same  strategic  result" 

-  Winston  Churchill 


k 

Performance  Risk 
Critical  Vulnerabilities 

r 


00 

World  Class 
Performance 


Process 

^  Enricfomenfc^^i/ 


Entente 

Powers 


End  State: 
Competitive 
Position 


Performance  Quality 
Competitive  Advantages 


Competitive  Advantage: 

use  excess  ships  against 
coastal  fortresses;  land 
troops  before  coast  is 
fortified  by  the  enemy 


Optimization^^ 
Sustainment  oo 


Efficacy  QQ 


Critical  Vulnerability: 
light  defenses  in  the 
Turkish  Dardanelles 


O 


Acceptance 


A  Mutually  Destructive  Market 

Each  side  became  operationally  efficient  in  trench  warfare,  but 
the  stalemate  wouldn't  break.  Losses  on  both  sides  were  heavy. 


The  British  Fleet  at  the  Dardanelles 


Case  Study 


The  1 967  Six-Day  War 

Assessing  Competitive  Position 


Performance  Risk 
Cnhcal  Vulnerabilities 

f 


End  State: 
Competitive 
Position 


t 

Performance  Quality 
Competitive  Advantages 

i 


oo 

World  Clast- 
Performance 

Israeli  Air  Force _ 

.  A  Process 
rich  merit 

n 

i.  V  vl  w  1  1  i  1  |l  |  V/  1  W 

w 

Refit  Cycle  Time  =  7  Vi 
Minutes 

O 

Optimization 

O 

Yield  =  4  sorties/day 

O 

sustainment 

o 

<D 

Efficacy 

O 

Accepts  nee 

Refit  Cycle  Time  >  2 
Hours 

Yield  =  2  sorties/day 


In  the  first  170  minutes,  300  out  of  340  Egyptian  aircraft  were  destroyed.* 

By  noon  of  the  2nd  day,  the  Egyptian,  Jordanian  and  Syrian  Air  Forces,  with  about  450  aircraft,  were 
destroyed...  As  were  most  of  the  18  airfields  in  Egypt.  Israel  lost  26  aircraft. 

^uThe  Six  Day  War/  Randolph  &  Winston  Churchill,  1967 


Case  Study 


The  1 967  Six-Day  War 

Assessing  Market  Value 


Necessity  &  Scarcity 
Emergent  Needs 


Market 

Value 


Value  Improvement 
Emerges  Strategy 


Performance  Risk  ^  '  Performance  Quality 

Critical  Vulnerabilities  position  Competitive  Advantages 


Israeli  Air  Force 

- ►  oo 

World  Clast- 
Performance 

Switches  Targets 

The  value  of  Israel's 

Process 
WE  n  rich  merit 

superior  strategy  in  refit 
cycle  time  demonstrated  a 

World  Class  capability  to 
switch  targets  and  meet  the 

Optimization 

emergent  need  to  destroy 
the  Jordanian  Air  Forces' 
capability  to  operate 

sustainment 

Efficacy 

Acceptance 

Defective  Strategic 
Communication 


*  The  Six  Day  War/  Randolph  &  Winston  Churchill 


Process  Enrichment3"  Boot  Camp 

Class  of  2008 


Congratulations  Graduates! 


Victor  Elias 

High  Performance  Technologies,  Inc. 

velias@HPTi.com 

(973)  724  -  4858 
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Mapping  Acquisition 
Requirements  from  Capabilities 
in  a  Net-Centric  Enterprise  - 
Creating  a  Capabilities 
Engineering  Framework 


Jack  Van  Kirk 
Ira  Monarch 


NDIA  1 1th  Annual  Systems  Engineering 
Conference  10/20-23 


Software  Engineering  Institute 


Carnegie  Mellon 


©  2008  Carnegie  Mellon  University 


Software  System  Acquisition  Problem  Areas 

Requirements  Always  High  on  the  List 


^-ACAX-LAcquisjlixxn-Rrograms  under  scrutiny  (GAO  04-393)  -  significant  issues  published 

•  Boehm  :  ‘Reasons  Why  Programs  Fail’  -  Inadequate  Requirements  a  major  causal  factor 

•  Sandish  Report  and  others:  Inadequate  requirements  source  of  cost  and  schedule  overruns 
and  performance  shortfalls 


Little  Evidence  of  Requirements  Engineering  in  place 


Project 
Management 
Best  Practices 

Skills 

Training 
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Architecture 

Requirements 

Interoperability 

Process 

DSB  2000  Report 
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* 
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Emerging  Benchmark 
Results 

* 

* 

* 

# 

* 

# 
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Classic  Requirements  Management 


Verification, 
Operations  & 
Maintenance 


Concept  and 
Architecture 
Development 


Fabrication  & 
Production 


Design 

Development 


Requirements 

Development 


Requirements 

Elicitation 


Organizations 
,  &  People  . 


Design-to 

Specs 


User 

CONOPS 


As-built 


Musts 

Wants 

Weight? 


Build-to 

Specs 


Verification 

Procedures 


Validation 

Plan 


Verification 

Plan 


Anomalies 


,1 

f 

V 
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As- 

deployed  ] 

As- 
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Solutions 

Expecta¬ 

tions 
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As  Operorions  Requested  It 


As  Procurement  Ordered  It 


As  Plont  Maintenance  installed  It 


As  Engineering  Designed  (t 


As  Accounting  Paid  For  It 


What  the  Soldier  Wanted! 


The  Capability  Turn  in  Requirements  Development: 
A  Domain-Centered  Approach 


Software  quality  in  digitized  systems  depends  on  how  well  the  software  represents 
and  is  responsive  to  the  domain  contexts  in  which  the  systems  operate. 

A  capability  driven  approach*  builds  on  domain  centered  approaches  -  capabilities 
are  defined  wrt  to  a  context  containing  multiple  domains. 

User-driven,  domain-driven  &  capability-driven  approaches  to  software  intensive 
system  acquisition  all  point  in  a  similar  direction  - 


The  voice  of  the  customer,  in  this  case  the  warfighter, 
must  be  heard  down  to  the  software  technologist. 


The  voice  of  the  software  technologist  has  to  be  heard 

by  the  warfighter 


*  Capability  driven  approaches  in  the  military  stem  from  the  Joint  Capabilities  Integration  and  Development  System 
(JCIDS)  created  by  the  Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS) 
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The  Capability  Turn  in  Requirements  Development: 
Difficulties 


In  the  US  military,  capability  driven  approaches  are  difficult  to  implement  due  to 

•  the  huge  numbers  of  people  involved  and  their  very  different  perspectives  (e.g., 
warfighter  vs.  bureaucrat  vs.  technologist) 

•  the  rapidly  changing  and  uniqueness  of  threats 

•  the  pace  of  information  technology. 

From  analysis  of  10s  of  1000s  of  Problem  and  Trouble  Reports  it  appears  that 
capability  driven  approaches  are  not  informing  the  software  as  well  as  they  could. 

-  Software  problems  are  not  stated  in  terms  of  capabilities  being  adversely  affected 

-  Software  solutions  do  not  refer  to  how  enablement  of  capabilities  can  be  improved 


page  6 


Overcoming  Difficulties  for  the  Capability  Turn: 

A  Framework  for  Capability  Engineering 


The  aim  of  Capability  Engineering  (CE)  is  to  meet  the  challenges  capability  & 
domain  driven  approaches  face. 

CE  is  the  mutual  formulation  of  joint  capabilities  and  acquisition  requirements  for 
multiple 

•  platforms 

•  systems/subsystems  that  work  with  or  in  these  platforms. 

CE  supports  traceability  and  validation  of  requirements  specifications  from 
capabilities 

The  Capability  Engineering  Framework  (CEF)  provides  knowledge  management 
support  for  CE. 

The  CEF  identifies,  annotates  and  organizes  exemplary  practices. 
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The  Five  Dimensions  of  CEF 


The  five  CEF  Dimensions  organize  and  document  support  for  “good  practices”  in 
capability  engineering: 

1.  Organization  -  the  infrastructure  of  virtual  organizations,  which  are  multiple 
organizations  using  both  on-line  and  face-to-face  interaction  in  an  integrated  fashion. 

2.  Process  -  the  production  of  work  products  and  ultimately  the  product  itself,  especially 
to  processes  that  are  inter-organizational. 

3.  Information  -  (a)  finding  patterns  of  information  through  text  and  data  mining; 
(b)  structuring  information  via  domain  &  quality  models  across  stakeholders; 
and  (c)  organizing  information  flow  to  support  building  and  validating  material 
solutions. 

4.  Evaluation  -  assuring  quality  of  both  product  and  process,  and  especially  the  tie 
between  the  two. 

5.  Learning  -  the  integration  of  evaluations  and  other  forms  of  feedback  at  the 
enterprise  level  (both  PEO  and  SoS  or  FoS)  into  actionable  improvements. 

Current  CEF  work  focuses  on  the  Information  dimension  in  support  of  Battle 
Command  (BC)  Capability  Portfolio  Management  (CPM). 
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Information  Dimension:  Benefits 


There  are  several  benefits  of  capability  &  domain  driven  BC  software  design. 

1.  Traceability,  and  therefore  validation,  of  multiple  software  systems  and  systems  of  systems 
is  facilitated. 

-  Currently,  traceability  is  missing  and  validation  is  reduced  to  verifying  mission  threads 

-  S  &T  opportunities  are  under  appreciated  because  of  insufficient  mutual  understanding  between 
warfighter  and  software  technologist 

2.  Composing  system  of  systems  to  enable  capabilities  that  none  of  the  systems  alone  can 
enable  will  be  better  understood. 

-  Current  capability  documents  provide  a  partial  picture  of  how  systems  can  or  should  fit  together 

-  There  is  no  common  ground  for  reasoning  about  system  composition. 

3.  Capability  Portfolio  Management  across  programs  in  a  PEO  and  across  PEOs  will  be 
facilitated. 
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The  Information  Dimension:  Sources 


In  order  to  represent  the  domains  guiding  capability  driven  software, 

•  sources  of  domain  expertise  and  information  have  to  be  tapped 

•  processes  for  domain  modeling  must  be  established. 

In  the  military,  much  of  the  expertise  is  written  down  in  the  form  of 

=>  1.  Joint  Capability  Areas 

2.  Concept  Documents 

3.  Doctrine 

4.  Capability  Documents  (ORDs,  ONS,  ICDs,  CDDs,  CPDs...) 

5.  Information  Support  Plans  (ISPs) 

6.  User  Functional  Descriptions 

7.  Problem  and  Trouble  Reports 

8.  Shortfalls  and  Warfighter  Outcomes 

9.  Exercise  After  Action  Reviews, 

Independent  Evaluation  Results 
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AfterAction  Reviews  Field 

Forums  Reports 


Modify  Doctrine 
Build  Whole  Product 

DOTMLPF 
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Compose,  Use  & 
Test  Systems 


War  Games 
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Exercises 

—  V 


st  Systen 
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Problem  and 
Change  Reports 


FORSCOM 

MACOMs 


CoCOMs 
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Joint  Capability  Area  Focus: 
Battle  Command  Capability  i 


Command  &  Control 

Organize 

Establish  &  maintain  unity  of  effort  w/  mission  partners 

Develop  Trust 

Estab  &  Cultivate  Rel  w  Msn  Partners 
Estab  &  Cultivate  Rel  w  Partner  Orgs 
Structure  organization  to  mission 
Define  structure 
Assess  Staff  Capabilities 
Delegate  Authority 
Identify  Capabilities  Needed 
I  ntegrate  Capabilities 
Estab  Commanders'  Expectations 
Foster  organizational  collaboration 
Estab  Collaboration  Policies 
Estab  Collaborative  Procedures 

Understand 

Organize  I  nformation 

Develop  Knowledge  and  Situational  Awareness 
Share  Knowledge  and  Situational  Awareness 
Planning 

Analyze  problem 
Analyze  Guidance 
Review  Rule  Set 
Review  Situation 
Determine  Need  for  Action 
Prepare  Estimates 
Apply  situational  understanding 
Assess  Available  Capabilities 
Evaluate  Environment 
Determine  Vulnerabilities 
Determine  Opportunities 
Develop  strategy 
Determine  Force  Readiness 
Determine  Resources 
Adapt  Strategy 
Align  Strategy 
Develop  Assumptions 
Develop  Objectives 
Determine  End  State 
Review  Existing  Plans 
Develop  courses  of  action 
Understand  Objectives 
Develop  Options 
Establish  Selection  Criteria 
Analyze  courses  of  action 
War  game  courses  of  actions 
Compare  courses  of  actions 


Decide 

Manage  risk 
Validate  Targets 
Formulate  Crisis  Assessment 
Provide  Friendly  Force  Combat  I  dentification 
Direct  Consequence  Management 
Select  actions 
Select  course  of  action 
Select  Plan 
Terminate 
Establish  rule  sets 
Establish  intent  and  guidance 
Establish  Priorities 
Establish  Standards 
Establish  Rule  Sets 
I  ntuit 

Recognize  Key  Triggers 
Modify  Actions 

Direct 

Communicate  intent  and  guidance 

I  ssue  Estimates 
I  ssue  Priorities 
I  ssue  Rule  Sets 
Provide  CONOPS 
Task 

Synchronize  Operations 
Synchronize  Execution  across  Phases 
I  ssue  Plans 
I  ssue  Orders 
Establish  metrics 
Establish  Performance  Measures 
Establish  Effectiveness  Measures 

Monitor 

Assess  compliance  with  guidance 
Assess  Employment  of  Forces 
Assess  Manner  of  Employment 
Assess  effects 
Assess  Battle  Damage 
Assess  Effects  of  Deception  Plan 
Assess  Munitions  Effects 
Assess  Performance 
Assess  Re-Engagement  Requirement 
Assess  Operational  Effects  of  Strategic  Communications 
Assess  achievement  of  objectives 
Assess  guidance 
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Joint  Capability  Area  Focus: 
Battle  Command  Capability  2 
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Processing  /  Exploitation  (CNE) 

Correlate 

Convert 

Exploit 

Analysis  &  Production 

Intel  Prep  of  Opnl  Environment 

Intel  Sptto  Situational  Understanding 

Indications  &  Warnings 

Intel  Spt  to  Targeting,  FP  &  IO 

Battle  Damage  Assessment 

Science  &  Technology 

Counter  Intelligence 

1  SR  Dissemination 

Environment 

Collect 

Analyze 

Predict 

Exploit 

Net-Centric 

1  nformation  T  ransport 

Switching  and  Routing 

Wireless 

Wired 

Enterprise  Services 

Core  Enterprise  Services 

Collaboration 

Mediation 

Discovery 

Messaging 

1  nformatbn  Sharing/ Computing 

Data  Storage 

Data  Processing 

COI  Services 

Positbn  Navigation  and  Timing 

Net  Management 

Optimized  network  functions  &  resources 

Deployable,  scalable  &  modular  networks 

Spectrum  Management 

Cyber  Management 

1  nformation  Assurance 

Secure  1  nformation  Exchange 

Ensure  Authorized  Access 

Protect  Data  and  Networks 

Monitor  IA  Status 

T rack  User  Actions 

Prevent  Network  Attack 

Protect  Data  from  Modification 

Respond  to  Attack  /  Event 

Detect  &  Respond  to  Attacks 

Detect  &  Respond  to  Event 
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USJFCOM  JS 


Capability  to  System  Mapping: 

Joint  Common  System  Function  List  (JFCOM-  JSIC) 


•  Mapping  systems  to  system  functions  enables  traceability  to  Joint  &  Army-wide  operational  capabilities 

•  The  Joint  Common  System  Function  List  (JCSFL)  is  cumbersome  &  manually  applied  by  JSFL  experts. 

•  Successful  mapping  may  be  facilitated  by  automated  support  that  could  leverage  the  JCSFL 
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•  Engage  with  PEOs  to  evaluate  current  proposed  JCSFL  mappings  &  viability  of  automated  support 

•  Proposed  manual  mappings  include  AMPS,  DCGS,  FBCB2,  FCS,  GCCS,  JWARN,  Prophet,  SaaS,  TAIS 


Capability  to  System  Mapping: 

Concept  Maps  &  Domain  Modeling 


Both  automated  and  interactive  analyses  will  be  performed  on  collections 
of  documents  chosen  from  each  information  source. 

Automated  content  analysis  will  produce  concept  maps  of  selected 
information  sources. 

Concept  maps  will  be  interpreted  and  aligned  to  the  extent  possible. 

The  aim  is  to  find  conceptual  links  among  maps  of  the  information  sources 
that  will  support  domain  modeling  of  BC  contexts. 

The  BC  context  currently  being  investigated  is  Army  Aviation. 

The  current  focus  is  to  align  BC  enabling  systems  as  specified  by  PEO 
Aviation  with  planning  capabilities  as  specified  by  TRADOC. 
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Methodology : 

Content  Analysis  &  Concept  Maps 


Semi-automated  content  analysis  uses  automated  text  analysis  tools  to  identify  recurring 
concepts  &  clusters  of  concepts: 

•  Concepts  are  synonyms  of  strongly  related  co-occurring  terms  identified  in 
automatically  generated  affinity  lists 


•  Concept  Clusters  are  collections  of  co-occurring  concepts 


—  more  strongly  related  to  each  other  than  to  concepts  in  other  clusters 

named  by  automatic  selection  of  the  concept  most  strongly  related  to  other  concepts  in  the 
cluster  —  — _ 


Concept  Clusters  are  represented  graphically  as  Venn  diagrams. 

•  concepts  labeling  dots  are  in  concept  clusters  represented  as  circles 


•  dots  can  be  linked  by  lines  whose  brightness  represents  frequency  of  co-occurrence 


•  dots  can  appear  in  the  overlap  of  two  (or  more)  circles 


•  circle  size  based  on  distribution  of  concepts  included  in  the  circle  (not  importance) 


-  brightness  represents  interconnectedness  of  concepts  in  the  circle 
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Content  Analyses  and  The  Role  of  Interpretation 


Map  overlays  can  delimit  groups  of  concepts  from  more  than  one  concept  cluster 
according  to  human  interpretation,  e.g.,  BC,  BC  enablers,  helicopters 


Interpretation  also  depends  on  posing  and  answering  specific  questions, 

•  Question:  Are  there  concepts  that  trace  back  from  documentation  of  BC 
software  intensive  systems  to  documentation  of  BC  capabilities? 

•  Traceability  Potential:  Route  and  its  role  in  BC  planning  is  one  such 
concept. 

The  maps  shown  require  additional  interpretation  in  collaboration  with 
combatants,  domain  experts,  requirements  and  capability  developers  and  testers. 
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Aligning  Concept  Maps: 

On  the  Way  to  Domain  Modeling 


Shared  Kernel  (e.g.,  route) 


Adopted  from  Eric  Evans,  Domain  Driven  Design,  Tackling  Complexity  in 
the  Heart  of  Software,  Addison  Wesley  Professional,  2003 


Joint  &  Army  Doctrine 


ONS,  AARs,  Gaps, 
Shortfalls,  Lessons  Learned 


ORDs,  Capability  Documents,  UFDs  &  ISPs 
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Interpreting  Route  in  Army  Aviation  Concept  & 
Doctrine 


Operations  Concept  (2008): 

•  Route  plays  a  role  in  BC  capabilities  enabled  by  software  intensive  systems 
and  is  used  in  Army  Aviation  operations 

•  More  specifically,  route  is  used  in  C2  planning  and  to  a  lesser  extent  in  other 
BC  activities  and  BC  enabling  systems 

•  Though  several  specific  helicopters  are  mentioned,  route  links  to  two  -  AH-64D 
&  ARH-70 

Operations  Doctrine  (2008  draft  2007): 

•  Route  plays  a  role  in  an  Aircraft’s  flight  &  C2  operations,  and  also  wrt  planning 

•  Route  &  planning  link  to  BC  concepts  but  are  somewhat  separated  from  BC 
discussion 

•  Route  links  to  discussion  of  specific  helicopters  -  not  the  specific  aircraft  but 
concepts  discussed  with  these,  e.g.,  radar,  infrared  systems  &  visualizing 
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Interpreting  Route  in  Army  Aviation  C2  Doctrine 
and  Planning  System  DFD 

C2  Doctrine  (2002): 

•  Route  plays  a  role  in  air  defense  operations  &  control  of  the  aircraft  in  airspace 

•  It  is  used  in  planning  and  A2C2  and  to  a  lesser  extent  in  the  command 
coordination  hierarchy 

•  Planning  is  within  the  BC  overlay  that  includes  concepts  of  BC  &  its  enablers 

•  No  mention  of  specific  helicopters 

Planning  System  Desired  Functions  Document  (2007) 

•  The  focus  is  on  route’s  role  in  planning  capability  &  the  aircraft’s  flight/mission 

•  Also  in  focus  are  information  systems  as  capability  enablers  and  Data  as 
rendered  in  charts 

•  The  overlay  of  BC  concepts  is  contained  in  the  Plan  concept  cluster,  as  is 
route 

•  Closely  related  overlays  specifically  refer  to  BC  enabling  (BCE)  software 
intensive  systems  &  helicopters 
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Analysis  of  Army  Aviation  BC  Documentation: 

Planning  System  STRs 


Planning  System  Development  STRs  (2008): 

• Route  is  thematic  and  consists  of  points  created  by  a  user  in  dialog  with  the 
software  modules  SAGE  &  AWE  manipulating  messages  &  files 

• Routes  are  imported  from  files,  created,  selected  and  displayed 

•Data  changes  and  changing  values  occur  and  are  linked  to  route 

•All  the  above  are  implicated  in  errors 


Planning  System  Post-Development  STRs  (2008): 

• Route  consists  of  points  graphically  displayed  in  dialog  with  SAGE,  though  change 
is  associated  with  route  not  data 


•Graphics  and  dialog  are  now  thematic 

•File,  message  and  user  are  most  associated  with  error. 


Imported  waypoints  are  now  closely  associated  with  route  as  is  Mission  Planning 
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Analysis  of  Army  Aviation  BC  Documentation: 

Planning  System  STRs-  Route  as  Domain  Concept 


The  Planning  System  STRs  are  not  capability  focused,  and  rather  given  to 
buttonology,  but  they  do  make  contact  with  BC  contexts  and  domains  through  route 
and  user. 

Route  is  a  domain  concept  that  needs  to  be  represented  via  domain  modeling  of  BC 
Aviation  contexts  informing  software  development,  acquisition  and  testing. 

We  have  shown  that  TRADOC  pamphlets,  doctrine  and  DFDs  could  be  utilized 
so  that  capability,  domain  and  user  centered  testing  has  impact  on  prioritizing 
maintenance,  refinement  and  evolution  of  systems. 

We  are  planning  meetings  with  combat  and  material  developer  domain  experts 
to  identify  more  concepts  like  route  that  will  be  sufficient  for  building 

•  domain  models  in  each  sphere  of  expertise 

•  aligning  the  models  in  the  Army  Aviation  BC  context 
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Summary: 

Establishing  Shared  Conceptual  Structures 

Operational  Military  Information  Flow 


Demand 


CALL 


Lessons 

Learned 


Capture.  *"^mafee 

Synthesize- 


Marir^LL 
After  Actton  Reviews  Field 

Forums  Reports 


Modify  Doctrine 
Build  Whole  Product 

DOTMLPF 


w 
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Thank  you  for  your  attention! 


For  further  information, 
please  contact: 


Jack  Van  Kirk, 

iack.vankirk@us.armv.mil 

256.955.0698 

or 

Ira  Monarch 

iam@sei.cmu.edu 

1.412.268.7070 


_  Software  Engineering  Institute  CarnegieMellon 


Mapping  Acquisition  Requirements  from 
Capabilities  in  a  Net-Centric  Enterprise  -  Creating 
a  Capabilities  Engineering  Framework|  NDIA 
Systems  Conference  10/21 
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Best  Practices  Clearinghouse: 

Making  Lessons  Learned 
Come  Alive  and  Be  Practical 


Defense  Acquisition  University 


Forrest  Shull, 

Fraunhofer  Center  Maryland 

NDIA  Systems  Engineering  Conference 

October  2008 
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Objectives 


■  Review  the  DoD  Acquisition  Best  Practices 
Clearinghouse  (BPCh)  approach  and  tool 

■  Describe  our  processes  for  working  with  both 
structured  and  unstructured  content 

>  And  raise  interest  in  submitting  your  own  content 

■  Discuss  some  of  the  emerging  priorities  and 
best  practices  we  are  finding 
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■■mi _ _ _ 
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The  DoD  Acquisition 
Best  Practices  Clearinghouse 


EME 


v  Certificate  Error  X  iooqle 


DoD  Acquisition  Best  Practices  Clearinghouse!  -  Windows  Internet  Explorer 


T  'v  https://bpch.dau.mil/Pages/default.aspx 


File  Edit  View  Favorites  Tools  Help 


*  *  m  DoD  Acquisition  Eiest  Practices  Clearinghouse! 


T  A  T  l^P^ge  -  Tools  - 


£?£  Best  Practices  da, 


Defense  Acquisition  University 


DAU  Homepage 
I  Need  T rain  ing 
Con  tin  nous  Lea  ming 
Knowledge  Sha  ring 
Performance  Support  ► 


-BPCh  Menti- 


Browse  Content  Views 
Filter  Content 
Submit  Content 
Feedback 
About  BPCh 


Done 


m _ ■ 

Home  |  DAU  | 

|  Contact  |  Site  Hap  |  FAQ  | 

Help 

|  Search  | 

Welcome  to  the  Acquisition  Best  Practices  Clearinghouse 


Quick  Search 


search 


Practices  that  have  the  most  evidence 

•  Software  Formal  Inspections 

•  Pair  Programming 
»  Trade  Studies 

•  Architectural  Reviews 

•  Integrated  Project  Data  Repositories  (IPDRs) 


Evidences  that  have  the  highesttru  stability  scores 

*  Advances  in  Software  Inspections 

•  An  Analysis  of  Defect  Densities  Found  Durina  Software  Inspections 


BPCh  Learning  Guides 


The  DoD  Acquisition  Best  Practices  Clearinghouse  (BPCh)  facilitates  the  selection  and 
implementation  of  systems  engineering  and  software  acquisition  practices  appropriate  to  the  needs 
of  individual  acquisition  programs. The  BPCh  uses  an  evidence-based  approach,  linking  to  existing 
resources  that  describe  howto  implement  various  best  practices.  These  linked  resources  also 
provide  descriptions  of  the  practical  results  (both  good  and  had)  of  applying  the  practices  in  various 
contexts,  from  which  users  can  learn  about  the  results  to  be  expected  in  their  environment.  All 
evidence  stored  is  also  contextualized,  so  that  users  will  he  guided  to  the  lessons  relevant  to  their 
program,  type  of  problem,  or  specific  situation. 


Guide  Links 

For  First-Time  Users  of  BPCh 
Contributing  Content  to  BPCh 
BPCh  Tutorials 


Explaining  Gold,  Silver  &  Bronze  Practice  Maturity 
Levels 


Understanding  the  BPCh  Vetting  Process 


Gold  Practices 

*  Pair  Programming 

•  Software  Formal  Inspections 


Practices  that  Reduce  Schedule 
*  Include  a  Requirements  Database  in  the  RFP 


Practices  that  Improve  Quality 

*  Pair  Programming 

*  Software  Formal  Inspections 

*  Software  Walkthroughs 


Acquisition  KM  Systems 


9  f 


BPCh 


V 


t *  Internet 


%  100%  - 


■O  ^  \  ©  Inbox  -  Micros... 

V  presentations  >  2008. 10  NDIA. . . 

fcl  iTunes 

2006 . 03 . 22  A . . ,  C  DoD  Acquisitio ...  '  < 

What  makes  BPCh  unique? 


Contents 


Intro  to  BPCh 


_  Processes  and 

examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


■  Not  all  best  practices  are  “best”  for  everybody 

>  Content  includes  descriptions  of  past  results  in  context, 
not  just  what  to  do 

>  Allows  context-sensitive  search  (show  me  just  the 
practices  that  programs  like  mine  have  used) 

>  Recommendations  built  on  evidence 


Pointers  to  existing  sites,  resources,  examples 
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Center  for  Experl  mental 
Software  Engineering 
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Overview  of  building  content 


•For  more  information  click  on  the  following  links: 


Practice 

Maturity 


Source 

Context 

Results 


Source 

Context 

Results 


Source 

Context 

Results 


Source 

Context 

Results 
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Definitions 


Contents 


Intro  to  BPCh 


_  Processes  and 

examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


■  A  practice  is: 

>  A  documented  activity  that  is  described  in  an 
actionable,  repeatable  way; 

>  A  description  of  howto  do  something,  not  a  general 
goal  of  what  to  do 

>  May  be:  A  process,  method,  technique,  standard... 

■  Evidence  about  a  practice: 

>  Is  a  description  of  an  experience  which  provides  a 
better  understanding  of  a  situation 

>  Similar  to  a  lesson  learned 

>  Composed  of: 

❖a  practice, 

❖a  context  and 
❖a  discernible  result. 
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Representing  Context 


Contents 


Intro  to  BPCh 


_  Processes  and 

examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


■  Any  piece  of  evidence  is  tagged  according  to 
where  it  was  drawn  from: 

>  Target  role  (acquirer,  developer) 

>  Domain  (warfighter,  business,  intelligence,  enterprise 
integration  environment) 

>  Criticality  level  (normal,  mission,  safety,  security) 

>  Integration  level  (software  application,  standalone 
subsystem,  platforms,  major  system,  system  of  systems) 

>  Environment  (military,  other  govt.,  industry,  academia) 

>  ACAT  level  (I,  IA,  II,  III) 

>  Lifecycle  phases  where  practice  used:  (Concept 
refinement,  Technology  development,  System 
development  &  demonstration,  etc.) 

>  Organizational  scope  (individual,  project,  program, 
organization,  enterprise) 
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United  States 


DEPARTMENT 


Contents 


Intro  to  BPCh 


Processes  and 
examples 


The  users* 
view 


How  can  I 
get  involved? 


BPCh  Content  Manager  and 
Subject  Matter  Experts  (SMEs) 


List  of 
priorities 


Topicl 

Topic2 

Topic3 

Topic4 

Topic5 


User- 

submitted 

Content 
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Leads  list 


Content 

Manager 


Structured,  e.g. 
Case  studies 
GAO  reports 
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SME  1 

i 

1 

L 

r 

►  SME  2 

i 
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Unstructured,  e.g. 

Guidebooks 
Program  reviews 
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Current  Priorities 


Contents 

—  Intro  to  BPCh 

^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


■  As  determined  by  Content  Advisory  Group,  input 
from  independent  review  teams,  conference 
feedback: 

>  Logistics 

>  Systems  Engineering 

>  Modeling  &  Simulation  (M&S) 

>  Program  Management 

>  System  Assurance 

>  Contracting 
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Example:  Air  Force  Institute  of  Technology  (AFIT) 
Case  Studies 


Contents 

—  Intro  to  BPCh 

^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


B?2  Sytlemi  L  i>gint«rir>g 
C Study 


utiiinc  vi'au  ii  mmjlkim 

i  \k ;iNf i  iii 
C.A.M  SIUTTV 
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Example:  AFIT  Case  Studies 


Contents 

—  Intro  to  BPCh 


^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


■  ■  ■ 


□  □  □ 


Identifying  practice  leads: 

>  AFIT  learning  principles’  explicitly 
identified  important  lessons  contributing 
to  success  /  failure  of  systems  analyzed 

♦♦♦Mostly  SE,  PM 

Creating  evidence: 

>  The  case  studies  provide  in-depth 
examination  of  a  particular  program  that 
could  be  mined  for  evidence 

Fleshing  out  practices: 

>  Working  with  AFIT  personnel  and  case 
study  analysts  to  provide  appropriate 
detail  about  the  practices. 
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Example:  AFIT  Case  Studies 


Contents 

—  Intro  to  BPCh 

^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


Example  results: 

>  New  /  Modified  Practices: 

❖  Invest  in  and  retain  core  engineers  and  staff 

❖  Integration  of  requirements  and  design  process 

❖  Effective  validation  and  verification  requires  a  firm 
requirements  baseline 

❖  Implement  technology  development  plan  when  technology 
spans  multiple  programs 

>  Existing  Practices: 

❖  Independent  Reviews 

❖  Work  Breakdown  Structure 

❖  Distributed  Work  Allocation 

❖  Architectural  Trade-off  Analysis  Method  (ATAM) 

❖  Systems  Engineering  Plan  (SEP)  Preparation  Guide 
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Example:  Program  Support  Reviews 


Contents 

—  Intro  to  BPCh 


^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


■  ■  ■ 


□  □  □ 
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Identifying  practice  leads: 

>  Conducted  a  brainstorming  session  with 
technical  experts  to  capture  trends, 
recurring  problems 


Creating  evidence: 

>  Reviewers  provided  insights  from  the 
programs  they  reviewed,  that  illustrate 
the  practices  they  discussed 


Fleshing  out  practices: 

>  Plan  to  conduct  follow-up  meetings  with 
the  programs  themselves  to  get  more 
detail  about  how  practices  were 
implemented 
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Example:  Program  Support  Reviews 


Contents 

—  Intro  to  BPCh 

^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


Example  practices: 

■  Include  requirements  database  in  Request  for  Proposal 
(RFP)  process 

■  Get  potential  bidders  to  comment  on  SRR  before  RFP 

■  Develop  system  engineering  plan  prior  to  RFP  release  and 
include  RFP 

■  Independent  cost  &  schedule  estimate 

■  Independent  reviews 

■  Establish  a  battle  rhythm  for  reports 

Integrated  Developmental  Test  /  Operational  Test  (DT/OT) 


Contents 

—  Intro  to  BPCh 

^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 
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Other  Emerging  Practices:  Logistics 


Contents 

—  Intro  to  BPCh 

^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


Performance-Based  Logistics  (PBL) 

>  Business  Case  Analysis 

>  Award  Contract 

>  Supply  Chain  Management 

>  Performance-based  agreements 

>  Resource:  DAU  Acquisition  Community  Connection  (ACC)  PBL 
toolkit 

■  Sustainment 

>  Technology  Insertion 

>  Software  Sustainment 

>  Item  Unique  Identification  (IUID)  /  Radio  Frequency 
Identification  (RFID) 

>  Independent  Logistics  Assessments 

>  Prognostics  &  Health  Management  and  Enhanced  Diagnostics 
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Other  Emerging  Practices:  M&S 


Contents 

—  Intro  to  BPCh 

^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 


Involve  Operational  Test  Authority  in  M&S  planning  to 
support  DT/OT  objectives 

■  Develop  M&S  plans  and  integrate  with  Test  Evaluation  and 
Management  Plan  (TEMP) 

■  M&S  reuse 

>  Based  on:  domain  info,  conceptual  model,  algorithms,  software 
components,  input  data  sets... 

■  Include  M&S  in  contractual  provisions 

>  Addressing:  representation  requirements,  data  rights,  M&S 
planning  and  documentation,  ownership  of  resources... 


Contents 

—  Intro  to  BPCh 

^  Processes  and 
examples 

_  The  users* 

view 

_  How  can  I 

get  involved? 
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What  the  User  Sees...  An  Example  Practice 


Browse  Content  Views 

Systems  Engineering  Plan 

Program  Requirements  ► 

Filter  Content 

CMMI  Acquisition  Module  (CMMI-AM) 

► 

Technical  Staffing  and  Organizational  Planning 

Submit  Content 

Career  Field 

* 

Technology  Maturation  and  Planning 

Feedback 

Software  Acquisition  Management 

Technical  Review  Planning 

About  BPCh 

Bj™|e  Practice  Name 

Integration  with  Overall  Program  Management 

Practice  Summary: 

Evaluations  of  the  tradeoffs  among  operational  capabilities,  functiona 
testing,  and  support  processes:  program  schedule:  and  lifecycle  cost 


Capabilities,  Requirements  and 
Concept(s)  of  Operation 


Other  Requirements  Linked  to 
the  Preferred  System  Concept 


Capabiliti 


Critical  Technologies 


Technology  Maturation  Cost/  Schedule 
Constraints 


Technology  Development  and 
Evolving  Acquisition  Strategy 


Bronze 

w 

# 


Practice  Name  :  Utility  Curve  Methodology 


Practice  Summary: 

A  common  methodology  used  to  perform  trade-off  analysis.  It  is  widely  used  for  cost  effectiveness  analysis  and  p 
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Maryland 


Bronze 

IIP 


Practice  Name  :  Requirements  Allocation  Sheet 
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What  the  User  Sees...  An  Example  Practice 


Practice  :  Software  Formal  Inspections 

Evidence  [11)  r  Resources  [2) 


Practice  Details 

Resources 

Summary 

Evidence  Name 

Rating 

Overall  Perception 

Quality  Experience  Report 

Criticality 

Primary  Benefit 

What  We  Have  Learned  about  Fiohtina  Defects 

8 

Via  interview 

Improved  Quality 

Applying  Proaram  Comprehension  Techniques  to 

Improve  Software  Inspections 

12 

Workshop  publication 

Reduced  Cost 

R&Dort  on  the  Loss  of  the  Mars  Climate  Orbiter 

Mission 

9 

Technical  report  (within  an 
organization  or  university} 

The  Empirical  Investiaationof  Perspective-Based 

Readina 

13 

b 

Archival  journal  publication  (e.g. 
IEEE  Transactions  on  Software 
Engineering) 

Normal 

Improved  Quality 

Com  Darina  the  Effectiveness  of  SoftwareTestina 

Strateaies 

14 

Archival  journal  publication  (e.g. 
IEEE  Transactions  on  Software 
Engineering) 

Improved  Quality- 

Space  Shuttle  Primary  Onboard  Software 
DeveloDment:  Process  Control  and  Defect  Cause 

An  a  Iv  s  is 

12 

Technical  report  (within  an 
organization  or  university} 

Safety  critical 

Improved  Quality 

Key  Lessons  in  Achievina  Widespread  Inspection 

Use 

17 

b 

Trade  journal  publication  (e.g. 
CrossTalk} 

Don't  know 

Reduced  Cost 

ExDerience  with  InsDection  in  Ultralarae-Scale 

DeveloDments 

18 

b 

Conference  publication  or  2nd- 
tier  publication  (EMSE,  IEEE 
Software,  CACM) 

Don't  know 

Reduced  Cost 

19  Archival  journal  publication  (e.g. 

IEEE  Transactions  on  Software 
EnaineerincfJ 


An  Analysis  of  Defect  Densities  Found  During 

Software  Inspections 


Improved  Quality 


Current  SMEs 


United  States 


Contents 


Intro  to  BPCh 


_  Processes  and 

examples 


The  users* 
view 


How  can  I 
get  involved? 


Systems  Engineering 

>  Dona  Lee  dona.lee@syseng-so.com 

michael.ucchino@afit.edu 


>  Mike  Ucchino 
Logistics 

>  Bruce  Hatlem 

>  Jill  Garcia 


bruce.hatlam@dau.mil 
jill.garcia@dau.mil 
Modeling  &  Simulation  (M&S) 

>  MikeTruelove  mike.truelove@syseng-so.com 
Program  Management,  System  Assurance,  Contracting 

>  None  participating 

Software  Acquisition  Management 

>  Larry  Baker  larry.baker@dau.mil 

>  Bob  Skertic  robert.skertic@dau.mil 
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How  can  I  participate? 


Contents 

—  Intro  to  BPCh 


_  Processes  and 

examples 

_  The  users* 

view 


How  can  I 
get  involved? 


Visit:  https://bpch.dau.mil 

■  Built-in  feedback  forms  in  the  application 

>  ...To  give  us  a  lead 

>  ...To  suggest  a  practice  we  should  have 

>  ...To  tell  us  your  experience  with  a  practice 

>  ...To  give  us  a  detailed  experience  report 

■  Ability  to  integrate  BPCh  with  in-house  best  practice  / 
lessons  learned  systems 


Fill  out  our  questionnaires... 

>  To  suggest  other  content 

>  To  volunteer  as  a  SME 


Fraunhofer  USA,  Inc 
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Questions? 


Contents 

—  Intro  to  BPCh 


_  Processes  and 

examples 

_  The  users* 

view 


How  can  I 
get  involved? 
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Feel  free  to  contact: 

Forrest  Shull 

fshull@fc-md.umd.edu 

301-403-8970 

or 

Mike  Lambert 
Michael.Lambert@dau.mil 
703-805-4555 
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List  of  used  abbreviations 


a  <n>i 


United  States 


Contents 

-  Intro  to  BPCh 


_  Processes  and 

examples 


The  users 
view 


How  can  I 
get  involved? 


ACC: 
ACAT: 
AFIT: 
BPCh: 
CoP: 
COTS: 
DAU: 
DT/OT 
DoD: 
IUID 
M&S 
OSD: 
PBL: 
PM: 
RFID 
SE: 
SMEs: 
SSR 

HZi  Fraunhofer  USA,  Tnc^ 


Acquisition  Community  Connection 

Acquisition  CATegory 

Air  Force  Institute  of  Technology 

(Acquisition)  Best  Practices  Clearinghouse 

Communities  of  Practice 

Components  Off  The  Shelf 

Defense  Acquisition  University 

Developmental  Test  /  Operational  Test 

U.S.  Department  of  Defense 

Item  Unique  Identification 

Modeling  and  Simulation 

Office  of  the  Under  Secretary  of  Defense 

Performance  Based  Logistics 

Program/Project  Manager 

Radio  Frequency  Identification 

Systems  Engineering 

Subject  Matter  Experts 

System  Requirements  Review 

Test  Evaluation  and  Management  Plan 


Center  for  Experimental 
Software  Engineering 
Mary  land 
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11th  Annual  Systems  Engineering  Conference 


Prognostics  to  Improve  Mission 
Readiness  and  Availability 


Sony  Mathew 

calce 

Center  for  Advanced  Life  Cycle  Engineering 
University  of  Maryland , 

College  Park,  MD  20742 
www. prognostics,  umd.  edu 


calce  Center  for  Advanced  Life  Cycle  Engineering 
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University  of  Maryland 
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Mission  Readiness  and  Availability 


•  Mission  readiness  of  a  product  is  a  measure  of  the  time  needed 
for  a  product  to  be  in  full  operational  state. 

•  Mission  readiness  is  directly  proportional  to  the  products 
availability  to  the  customer. 

•  Availability  is  the  probability  that  a  product  will  in  operational 
state  at  a  given  time. 

Availaility  =  Uptime  /(Uptime  +  Downtime) 

•  Lower  the  down-time,  higher  will  be  the  availability. 

•  Product  maintenance  and  logistics  play  a  major  role  in  ensuring 
more  availability  and  better  mission  readiness  of  the  product. 


calce  Center  for  Advanced  Life  Cycle  Engineering 
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University  of  Maryland 
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Useful  Terms 


•  Health:  A  product’s  health  is  the  general  state  of  the  product 
with  respect  to  the  expected  normal  operating  condition. 

•  Health  monitoring:  a  process  of  measuring  and  recording  the 
extent  of  deviation  and  degradation  from  a  normal  operating 
condition 

•  Prognostics:  the  process  of  predicting  the  future  health  of  a 
product  by  analyzing  the  recorded  deviation  or  degradation. 

•  CBM  (Condition-Based  Maintenance):  is  a  preventive  and 
predictive  approach  to  maintenance  based  upon  the  evidence  of 
need. 
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Condition  Based  Maintenance  (CBM) 

The  objective  of  CBM  is  to  assess  a  product’s  health  during  operation 
and  determine  if  and  when  maintenance  is  needed. 
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Outcomes  of  Maintenance  Decisions 


Corrective 

Unanticipated 

Failure 

•  Hazardous 

•  Costly 

•  Unscheduled 
maintenance 


Predetermined  CBM 


Regular 

Maintenance 

•  Must  inspect, 
repair  or  replace 
after  fixed  time  or 
operational  interval 

•  Can  be  costly 

•  Can  induce  failures 

•  Increased  down¬ 
time 


Health 

Monitoring 

•  Maintenance  is 
forecasted 

•  Continuous  monitoring 
of  health  can  decrease 
down-time 

•  Product  sustainment, 
and  re-use  options  can 
be  determined 
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Prognostics  for  CBM 

•  One  of  the  key  enablers  of  CBM  is  the  development  of 
the  PHM  technology. 

•  PHM  assesses  and  quantifies  the  extent  of  deviation  or 
degradation  from  an  expected  normal  operating 
condition. 

•  A  symptom  of  impending  failure  or  anomalous 
behavior  can  be  identified  with  the  aid  of  health 
monitoring  and  prognostics  techniques. 

•  Knowledge  of  prognostic  distance  allows  informed 
logistics  and  maintenance  decisions. 
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Why  PHM? 


•  Provide  an  early  warning  of  failures 

•  Forecast  maintenance  as  needed:  avoid  scheduled 
maintenance  and  extend  maintenance  cycles  [condition 
based  maintenance] 

•  Predict  the  product’s  reliability 

•  Assess  the  potential  for  life  extensions 

•  Provide  efficient  fault  detection  and  identification, 
including  evidence  of  “failed”  equipment  found  to 
function  properly  when  re-tested  (no-fault  found). 

•  Improve  future  designs  and  qualification  methods 

•  Reduce  amount  of  redundancy 
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CALCE  Approach 


Failure  Modes  Mechanisms  and  Effects  Analysis  (FMMEA) 
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CALCE  PoF  based  PHM  Methodology 


Material 
properties  and 
product 
geometries 

Estimated  life 
cycle  loading 


Maintenance 

records 


FMMEA 


Define  item  and  identify 
elements  and  functions 
to  be  analyzed 


Identify  potential  failure 
modes 


Identify  potential  failure 
causes 


Identify  potential  failure 
mechanisms 


Identify  failure  models 


Prioritize  the  failure 
mechanisms 
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operating  loading 


Conduct  data 
reduction  and  load 
feature  extraction 
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Predicting  Remaining  Life  Based  on 
Physics  of  Failure  (1/2) 


Monitored 
environmental  and 
operating  conditions 
of  test  board 


Simplified  data 
^  (e.g.,  data 

reduction,  and 
cycle  counting) 


Performed  physics-  _ k  Obtained  the 

of-failure  based  ' — V  remaining  life 
stress  and  damage 
assessment 
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Estimated  Remaining  Life  (days) 


Predicting  Remaining  Life  Based  on 
Physics  of  Failure  (2/2) 
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Remaining  Life  Assessment  of  NASA  Space  Shuttle 
Remote  Manipulator  System  (SMRS)  Electronics 


The  SRMS  is  used  to  place  satellites, 
space  station  equipment  and  other 
payloads  in  orbit.  The  first  SRMS 
flew  on  the  space  shuttle  mission 
STS-2  in  November  1981. 

By  using  the  existing  sensor  data, 
along  with  inspection  and  physics-of- 
failure  software  analysis,  it  was  found 
that  there  was  little  degradation  in  the 
electronics  and  they  could  be  expected 
to  last  another  20  years. 
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Army  AMS  A  A  -CALCE  Project 
Two  Year  Demonstration  System 


•  The  objective  of  this  project  was  to  demonstrate  predictive  capabilities  for 
the  remaining  life  of  electronic  components  mounted  in  military  vehicles. 

•  The  project  centered  around  exposing  test  boards  with  electronic 
components  mounted  on  them  to  on  and  off  road  terrain. 

•  Field  failures  agreed  quite  well  with  the  predicted  failure  using  the 
monitored  PWB  strain  and  existing  CALCE  failure  models. 
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Considerations  for  Data-Driven  PHM 
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Case  Study:  Data  Driven  Approach 


•  Computers  are  complex  electronics  systems  and  can  be  used  as  a  test  vehicle  for 
developing  robust  prognostics  methodologies. 

•  A  baseline  was  generated  using  10  new  computers. 

•  A  total  of  72  experiments  were  conducted. 

•  Duration  of  data  collection  at  each  setup  was  approximately  three  hours. 

•  Environmental  conditions 

1 .  5 °C  with  uncontrolled  Relative 
Humidity 

2.  25°C  with  55%  RH 

3.  25°C  with  93%  RH 

4.  50°C  with  20%  RH 

5.  50°C  with  55%  RH 

6.  50°C  with  93%  RH 

•  Usage  Levels 

1.  LI:  Benign 

2.  L2:  Low 

3.  L3:  Medium 

4.  L4:  High 

•  Three  Power  Settings 

•  Parameters  identified  for  health 
monitoring 

-  Device  information 

•  fan  speed,  LCD  brightness 

-  Thermal  information 

•  CPU  temp,  motherboard  temp, 
graphics  card  temp 

-  Performance  management 
information 

•  %CPU  usage,  %C1,  %C2,  %C3, 
%  CPU  throttle 
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MD  value 


Comparison  of  Mahalanobis  Distance  (MD) 
Values  for  Normal  and  Abnormal  Systems 


=tli  data  point 


Stats  (Model  A) 

Normal 

Abnormal 

Mean  of  MD 

0.83 

10.72 

Std.dev  of  MD 

1.16 

3.13 

The  data  from  the  10  new 
computers  used  to  form  the 
baseline. 

Utilizing  the  correlations 
between  the  measured 
parameters  MD  reduces  the 
multivariate  data  to  a 
univariate  data. 

An  NTF  computer  (Abnormal) 
was  tested  and  the  same 
parameters  were  recorded  as 
for  the  baseline  computers. 

The  MD  values  for  the 
Abnormal  system  showed 
faulty  behavior  at  time  zero. 
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Percentage  of  data 


Comparison  of  Histogram  of  MD  Values 


•  Test  computer  shows  different 
distribution  of  MD  values  as 
compared  to  baseline  computer 

•  This  demonstrates  the  test  computer 
has  different  signature 


•  3 -parameter  lognormal  distribution 
fit  for  the  baseline  MD  value  and 
more  than  95%  data  is  covered  by 
the  distribution 
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Principal  Component  Analysis 

•  Principal  Components  Analysis  (PCA)  is  used  in  a 
wide  array  of  applications  to  reduce  a  large  data  set  to 
a  smaller  one  while  maintaining  the  majority  of  the 
variability  present  in  the  original  data. 

•  Two  statistical  indices,  the  Hotelling  Squared  (T2)  and 
squared  prediction  error  (SPE)  are  used  in  the  PCA. 

•  The  SPE  statistic  is  related  to  the  residuals  of  process 
variables  and  is  a  reliable  indicator  to  a  change  in  the 
correlation  structure. 

•  The  Hotelling  T2  score  measures  the  Mahalanobis 
Distance  from  the  projected  sample  data  point  to  the 
origin  in  the  signal  space  defined  by  the  PCA  model. 


calce  Center  for  Advanced  Life  Cycle  Engineering 


18 


University  of  Maryland 

Copyright  ©  2008  CALCE 


Projection  Pursuit:  Analysis  Results  -  T2 


•  Tested  a  computer 
showing  abnormal 
behavior  against  a 
baseline. 

•  From  the  T2  analysis, 
test  computer  shows  a 
distinction  from  the 
baseline  data 

•  The  contribution  plot 
identifies  the  fan  speed 
as  the  dominant 
parameter  that 
contributes  to  the  shift 
from  the  baseline. 
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Projection  Pursuit:  Analysis  Results  -  SPE 


•  The  SPE  feature 
also  classifies  the 
test  computer  as 
different  from  the 
baseline. 

•  All  temperatures 
are  dominant  in  the 
residual  space  and 
are  identified  as  the 
influencing  factors 
for  the  fan  speed. 


0  500  1000  1500  2000  2500  3000  3500  4000  4500  5000 

Observation  # 
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Prognostics  Health  Monitoring  Enabled  Logistics 

Decisions  for  Aircraft  Carrier 


Wireless  transfer  of 
mission  usage  data 
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Conclusions 


Prognostics  using  approaches  including  PoF  based  life 
consumption  monitoring,  data  trending  and  analysis,  and  use 
of  canaries  can  be  achieved. 

Prognostics  and  health  monitoring  provides  advanced 
warning  of  failure  or  abnormal  behavior  and  thereby  helps 
determine  the  mission  readiness  and  availability  of  the 
product. 

Assessment  of  remaining  life  helps  drive  the  cost  effective 
logistics  decisions. 

Condition  based  maintenance  can  be  implemented  with  the 
help  of  health  monitoring  and  prognostics  technologies. 


calce  Center  for  Advanced  Life  Cycle  Engineering 


22 


University  of  Maryland 

Copyright  ©  2008  CALCE 


CALCE 


CALCE  founded  in  1987  is 
dedicated  to  providing  a 
knowledge  and  resource  base  to 
support  the  development  and 
sustainment  of  competitive 
electronic  products  and  systems 

Focus  areas: 

•  Physics  of  failure 

•  Design  for  reliability 

•  Accelerated  testing 

•  Qualification 

•  Supply  chain  management 

•  Obsolescence 

•  Prognostics 

Personnel: 

21  research  faculty 
6  technical  staff 
60+  PhD  students 
30+  MS  students 
1 1  visiting  scholars 


EPS 

Consortium 

40-45  companies 
Pre-competitive  research 
Risk  assessment, 
management,  and 
mitigation  for  electronics 


Education 

MS  and  PhD  EPS 
program 

International  visitors 
Web  seminars 
Short  courses  for 
industry 


Research 
Contracts 

•  Larger  programs 
Reliability  simulator  (KIM,  Korea), 
Remaining  life  assessment  (Air 
Force),  Tin  whisker  mitigation  (Navy,, 
TMTI),  Part  obsolescence  (NSF, 
NAVSEA),  Design 
refresh  planning  (Army 
CECOM,  DML) 


CALCE 

Center  for  Advanced 
Life  Cycle  Engineering 


Standards 

•  IEEE  1332 

•  IEEE  1413 
•IEEE  PI 624 

•  GEIA 

•  IPC 


PHM 

Consortium 

Pre-competitive  research 
Research  in  fundamental 
methodologies  to  develop 
and  implement  prognostics , 
and  health  management 
systems. 


Lab 

Services 

Small  jobs 
Fee-for-service 
Proprietary  work 
Use  of  CALCE  Tools  & 
Methods 

Turnkey  capabilities 
“Fire-fighting” 
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CALCE  Research  Focus  in  PHM 


•  Developing  the  capability  to  learn  from  data,  detect  changes  in 
real-time  and  predict  the  future  performance  of  electronic 
systems. 

•  Integrating  the  center’s  expertise  in  reliability  and  physics  of 
failure  (PoF)  of  electronic  components  into  hybrid  data  driven 
models  for  autonomous  system  prognostics  and  diagnostics. 

•  Researching  and  developing  prognostic  and  health 
management  technologies  that  will  enable  autonomous  fault 
diagnostics  and  prognostics  in  electronic  systems  such  that 
reliability  mitigations  can  be  implemented. 
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PHM  Book 


Overviews  the  concepts  of  PHM  and  the 
techniques  being  developed. 

Discusses  the  state-of-the-art  in  sensor 
systems. 

Discusses  the  various  data  driven/ 
statistical  models  and  algorithms. 
Discusses  the  physics-of-failure  based 
prognostics  approaches. 

Overview  of  the  implementation  costs 
and  return  on  investment  (ROI). 
Provides  a  roadmap  based  on  the  current 
challenges  and  opportunities  for  research 
and  development  of  PHM,  and 
Discusses  the  activities  of  the  major 
players  in  the  prognostics  research  field, 
including  companies,  academia  and 
government  organizations. 
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Agenda 


BAE  SYSTEMS 


*  Ground  Combat  Vehicle  Capabilities 

*  Levels  of  Maintenance 

*  Diagnostic  Technology  Evolution  -  past  and  future 

*  Prognostics  Definition 

*  Diagnostics  Concept  Design  and  Decomposition 

*  Possibilities  for  Enhancement 

-  Unit  Level  Diagnostics  (8) 

-  Direct  Support  Diagnostics  (5) 

-  Unit  Level  Diagnostics  &  Prognostics  (1) 

-  Prognostics  (3) 
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Ground  Combat  Vehicle  Overview 
-  Vehicle  Capabilities 


BAE  SYSTEMS 


•  Level  of  Technology  in  capabilities  typical  of  Ground 
Combat  Vehicles 

-  Mobility 

-  Lethality 

-  Communication 

-  Survival 

-  Transport 
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Vehicle  Capabilities  -  Mobility 


BAE  SYSTEMS 


•  Major  components 

-  Turbocharged  or  supercharged  reciprocating  diesel  engine 

-  Hydraulically  controlled  automatic  transmission 

-  Other  loads  -  hydraulic  pumps,  pneumatic  pumps, 
refrigeration  compressors,  direct  drive  engine  compartment 
cooling  fans,  electrical  generators,  and  the  supercharger 
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Vehicle  Capabilities  -  Lethality 


•  Capabilities  Provided 

-  target  sighting 

-  weapon  pointing 

-  ammunition  management 

-  round  discharge 

•  Technology  Evolution  -  target  sighting 

-  hard-mounted  passive  telescope  with  elevation  axis  adjustment 

-  Remote  superelevation  adjustment 

-  Electronic  measurement  of  target  range 

-  Coupling  target  range  measurement  to  superelevation  adjustment 

-  Imaging  of  other  than  visible  wavelengths 

-  Rasterized  video  imagery  to  permit  display  on  conventional  CRTs 
and  emerging  flat  panel  displays 

-  Remote  viewing  at  selected  crew  workstations 
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Vehicle  Capabilities  -  Lethality 

continued 


BAE  SYSTEMS 


•  Technology  evolution  -  weapon  aiming 

-  Manual  operation 

-  Hydraulics,  reducing  gunner  workload 

-  Electrical  as  power  electronics  became  more  capable 

-  Rate  commanded  directors 

-  Analog  servos  allowed  combining  the  operator  command 
with  an  inertial  gyro  input  yielding  inertial-stabilization 

-  Digital  servos  made  inclusion  of  other  battlefield  factor 
corrections  easier  to  implement,  reducing  the  gunner’s 
workload  again 

*  Technology  Evolution  -  Weapon  Control 

-  Mechanical  recharge  on  recoil 

-  Electronic  monitoring  and  control  managing  feeders  and 
improving  gunner  convenience  and  safety 
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Vehicle  Capabilities  -  Survival 


BAE  SYSTEMS 


•  Redundancy 

•  Battle  damage  protection 
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Vehicle  Capabilities  -  Not  Explored 


BAE  SYSTEMS 


•  Communication 

•  Transport 
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Supportability  - 
Current  Levels  of  Maintenance 


BAE  SYSTEMS 


*  Unit  Level  (Organizational  Level)  -  the  Motor  Pool 

*  Direct  Support  (Intermediate  Level) 

*  Depot  (typically  the  manufacturer) 
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Diagnostic  Technology  Evolution 


•  Before  1981 

-  Multimeters 

•  Vehicle  Schematics 

-  Vehicle  Test  Meter  (STE-ICE) 

•  Automotive  Diagnostic  Connector  Assemblies  &  Transducers 

•  Technical  Manual  troubleshooting 

•  1981 

-  Controllable  Interface  Box  (STE-M1/FVS) 

•  Weapon  System  Diagnostic  Connector  Assemblies 

•  Maintainer-augmented  fault  isolation 

•  Fault  isolate  to  single  LRU 

•  Matured  over  next  several  years  with  data  collected  in  Production 

-  Direct  Support  Electrical  System  Test  Set 

•  Replicates  vehicle  interfaces  on  bench 

•  Fault  isolate  to  single  SRU 

•  Matured  over  next  several  years  with  data  collected  in  Production 

•  1985 

-  Weapon  aiming  subsystem 

•  Conversion  to  digital  enabled  built-in  fault  isolation  routines 

•  Accessible  via  plug-in  terminal 

•  2001 

-  Turret  upgrade 

•  Systemic  BIT  requirement 

•  Fault  isolation  performed  by  main  system  computer 

•  Capabilities  available  statused  to  Commander 

•  Degraded  Modes 

•  Improvements  to  Direct  Support  testing 
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Diagnostic  Technology  -  Future 


•  Near  term 

-  LRUs  subject  to  obsolescence  redesign  include  BIT  and  Fault 
Isolation  to  the  SRU  level,  when  possible  based  solely  on 
monitoring  internal  LRU  behavior  only 

-  Results  saved  to  persistent  memory  and  made  available  to  a  plug¬ 
in  terminal,  making  the  Direct  Support  plug-in  test  equipment  that 
fault  isolates  to  SRUs  is  no  longer  required  for  those  LRUs. 

•  Longer  term 

-  Continue  to  include  BIT  and  Fault  Isolation  to  the  SRU  level  in 
LRUs  subject  to  obsolescence  redesign 

-  Include  system  wide  enhancements  so  that  LRUs  external 
behavior  can  be  stimulated  and  sensed  and  the  results 
communicated  so  the  LRU  is  able  to  react  to  external  observations 
and  perform  a  more  complete  fault  detection  and  isolation 

-  The  results  are  saved  to  persistent  memory  and  made  available  to 
a  plug-in  terminal. 
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Prognostics  -  A  Definition 


BAE  SYSTEMS 


•  Implementation:  Prognostics  requirements  are 
beginning  to  creep  into  contemplated  and  funded 
efforts,  but  still  as  a  placeholder 

•  Purpose:  To  estimate  remaining  useful  life  when  that 
life  is  nearing  its  end 

•  Technical  Requirement:  Predict  when  end-of-life  will 
occur  within  the  next  mission,  or  the  period  of  time 
the  vehicle  is  away  from  the  motor  pool.  More 
advance  warning  is  needed  if  the  replacement  part  is 
not  on  hand.  Obviously,  the  duty  cycle  of  each 
prognosed  component  is  critical  in  determining 
remaining  life  in  units  of  vehicle  power-on  time. 
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Diagnostics  Concept  Design 
and  Decomposition 


BAE  SYSTEMS 


•  Initial  diagnostics  concept  work  should  entail 

-  assessing  allocated  realizable  MTBF 

-  projected  mission  reliability 

-  development  and  unit  production  cost,  weight,  and  volume 

•  When  completed,  that  diagnostics  concept  work  should  result  in 

-  definitions 

-  requirements 

-  standardized  interfaces 

-  implementation  suggestions 

•  Then  the  emerging  system  and  subsystem  design  concepts  can  evolve 
to  include 

-  appropriate  partitioning  between  the  tactical  and  diagnostic  functions 

-  appropriate  level  of  inherent  fault  detection  algorithms  and  hardware  to  meet 
the  fault  isolation  requirement  and  intended  support  interface 

•  The  following  are  just  a  few  examples  of  capabilities  that  can  be  realized 
with  early  availability  of  diagnostics  requirements  and  concepts. 
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Possibilities  to  Enhance 
Unit  Level  Diagnostics 


BAE  SYSTEMS 


•  Minimize  suboptimal  compliance  with  requirements  by  enabling 
planning,  design,  and  review  of  compliance  early  in  the  subsystem 
design  cycle. 

•  Assure  that  pass/fail  limits,  algorithms,  and  crew/operator  messages 
are  updatable  separately  from  the  tactical  software,  so  diagnostics 
maturation  can  follow  an  independent  path  from  tactical  anomaly 
resolution  and  feature  addition. 

•  Characterize  abnormal  behavior  down  to  the  chip  level. 

•  Architect  intrusive  tests  such  that  they  may  be  executed  without 
affecting  the  in-vehicle  operation  of  the  electronics  assembly. 

•  Improve  LRU  interface  integrity  fault  detection  via  boundary  scan  at 
the  LRU’s  system  interface. 

•  Include  the  ability  to  tailor  diagnostic  pass/fail  limits  conditionally  to 
minimize  false  alarms  and  nuisance  trips  based  on  vehicle  mode  of 
operation. 

•  Include  LRU  degraded  modes  (such  as  reduced  processor  power 
consumption)  to  compliment  system  level  degrade  modes. 

•  Include  tests  of  system  interconnect  media  in  selected  LRUs  to  detect 
and  localize  breaks,  degradation,  and  missing  terminators. 


©  2008,  BAE  Systems  Land  &  Armaments  L.P.  All  Rights  Reserved 


14 


Possibilities  to  Enhance 
Direct  Support  Diagnostics 


BAE  SYSTEMS 


•  Include  an  LRU-level  persistent  memory  to  log  timestamped  pass-to-fail  and 
fail-to-pass  transitions  in  conjunction  with  data  potentially  important  to  a  root 
cause  analysis  (input  voltage,  internal  temperature,  value  of  analog  inputs, 
processor  load,  memory  utilization,  etc.)-  This  supports  bench  level  repair  and 
engineering  root  cause  analysis  of  failures. 

•  Include  a  standardized  interface  from  the  LRU  to  bench  power  and  a  USB  or 
other  standardized  serial  interface  port  to  enable  a  general  purpose  computer 
to  offload  the  fault  detection  log  and  fault  isolation  results,  manage  the 
persistent  memory,  and  optionally  accept  software  updates  for  the  LRU  or  for 
the  entire  vehicle. 

•  Allow  the  system  to  augment  LRU  interface  fault  detection  -  with  results 
reported  to  the  LRU  for  storage  in  it’s  persistent  memory  for  bench  level  repair. 

•  Include  sufficient  system  level  redundancy  and  partitioning  to  support 
reconfiguration  to  maintain  full  capability  or  introduce  degraded  modes  in  the 
presence  of  faults.  Examples  are  maintaining  full  capability  via  alternate 
processing  and  communication  resources,  degradation  by  invoking  less 
automated  capabilities,  reducing  Crewstation  access  to  capabilities,  etc. 

•  Combine  manufacturing  test  requirements  with  system  and  LRU  test 
requirements  and  satisfy  with  a  single  solution. 
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Possibilities  to  Enhance  both 
Unit  Level  Diagnostics  and  Prognostics 
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•  If  sensor  requirements  for  diagnostics/prognostics 
differ  from  those  for  tactical  operation,  select 
sensors  suitable  for  meeting  all  requirements. 
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Possibilities  to  Enhance 
Prognostics 


BAE  SYSTEMS 


*  The  follow-on  to  an  agreement  of  additional 
transducers  required  for  prognostics  is  to  implement 
the  interfaces  and  reserve  processing  power 
required  to  detect  degradation  (this  approach  may 
involve  high  frequency  characterization  of 
mechanical  systems  to  determine  degradation). 

*  Consider  including  board-resident  test  software  to 
track  component  degradation  over  time. 

*  Include  chip-level  monitoring  of  temperature  and 
input  current  if  deemed  pertinent  for  prognostics. 


©  2008,  BAE  Systems  Land  &  Armaments  L.P.  All  Rights  Reserved 


17 


BAE  SYSTEMS 


Open  Architecture  in  Electronics  Systems 


23  October  2008 
Bruce  Bardell,  Technical  Fellow 
Bradley  Chief  Architect 
BAE  Systems 


©  2008,  BAE  Systems  Land  &  Armaments  L.P.  All  Rights  Reserved 


1 


Open  Architecture,  10  years  ago 


BAE  SYSTEMS 


*  Goals  of  “Open  Architecture” 

-  to  guarantee  the  previously-developed  military  subsystems 
were  usable  in  platforms  under  development 

-  new  subsystems  being  developed  for  use  in  new  platforms 
could  be  reused  in  future  platforms 

*  How  is  “future  platform  reuse”  supported? 

-  Maximize  use  of  Industry  Standards 

-  Minimize  custom  design  content  of  any  “Vetronics”  item 
that  has  a  commercial  equivalent 

*  How  well  did  this  work?  See  next  slide. 


Vetronics  is  vehicle  computer  resources, 
vehicle  busses,  peripheral  electronics  such  as  sights, 
human/machine  interface  boxes,  electronic  GFE,  etc. 
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Open  Architecture,  10  years  ago 
What  is  “Open”  -  What  was  the  Result? 
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Standard  28  VDC  Vehicle  Power 
Source 

Standardized  command  /  response  & 
communication  busses,  such  as 
MIL-STD-1553B  &  Ethernet 

Standardized  interfaces,  such  as 

VGA,  RSI  70 

RS-232,  RS422,  RS423 

Standard  backplane,  such  as  VME 

Use  of  “Middleware”  OE  to  permit 
computer  HW  update 

Standard  form  factor  circuit  cards 
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Open  Architecture,  10  years  ago 
What  is  “Open”  -  What  was  the  Result? 
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Standard  28  VDC  Vehicle  Power 
Source 

Compatible  with  most  GFE 

Standardized  command  /  response  & 
communication  busses,  such  as 
MIL-STD-1553B  &  Ethernet 

Electrical  compatibility,  but  must 
comply  with  GFE  ICD 

Standardized  interfaces,  such  as 

VGA,  RSI 70 

Electrical  and  waveform  compatibility 

RS-232,  RS422,  RS423 

Electrical  compatibility,  but  must 
comply  with  GFE  ICD 

Standard  backplane,  such  as  VME 

Basic  compatibility,  but  VME  spec, 
includes  a  “custom”  connector 

Use  of  “Middleware”  OE  to  permit 
computer  HW  update 

No  Application  impact  when 
obsolescence  redesign(s)  introduced 

Standard  form  factor  circuit  cards 

No  benefit.  Function  density, 
environmentals,  and  “custom” 
connector  required  custom  CCAs. 

©  2008,  BAE  Systems  Land  &  Armaments  L.P.  All  Rights  Reserved  4 


Wikipedia  Definition,  21  Oct  2008 
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•  Open  architecture  is  a  type  of  computer  architecture  or  software 

architecture  that  allows  adding,  upgrading  and  swapping 
components.  For  example,  the  IBM  PC  has  an  open 
architecture,  whereas  the  Amiga  500  home  computer  had  a 
closed  architecture,  where  the  hardware  manufacturer  chooses 
the  components,  and  they  are  not  generally  upgradable. 

•  (Deleted  definition  that  relates  to  Architectural  Design  of 
Buildings) 

•  Open  architecture  allows  potential  users  to  see  inside  all  or 
parts  of  the  architecture  without  any  proprietary  constraints. 
Typically,  an  open  architecture  publishes  all  or  parts  of  its 
architecture  that  the  developer  or  integrator  wants  to  share. 
The  open  business  processes  involved  with  an  open 
architecture  may  require  some  license  agreements  between 
entities  sharing  the  architecture  information. 
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21  Oct  2008  Wikipedia  Definition,  parsed 
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•  Characteristics  of  Open  architecture 

-  allows  adding,  upgrading  and  swapping 
components 

*  For  example,  the  IBM  PC  has  an  open  architecture, 
whereas  the  Amiga  500  home  computer  had  a  closed 
architecture,  where  the  hardware  manufacturer  chooses 
the  components,  and  they  are  not  generally  upgradable. 

-  allows  potential  users  to  see  inside  all  or  parts  of 
the  architecture  without  any  proprietary 
constraints 

*  Typically,  an  open  architecture  publishes  all  or  parts  of 
its  architecture  that  the  developer  or  integrator  wants  to 
share.  The  open  business  processes  involved  with  an 
open  architecture  may  require  some  license  agreements 
between  entities  sharing  the  architecture  information. 


©  2008,  BAE  Systems  Land  &  Armaments  L.P.  All  Rights  Reserved 


6 


Building  on  the  Wikipedia  21  Oct 
2008  Definition  (1  of  2) 
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Wikipedia 

Possible  Manifestation  in 

Vehicle  Design 

How  does  Vehicle  Design  Enable? 

Allows 
adding, 
upgrading 
and  swapping 
components 

Uses  standard  power  levels 

Meet  Military  standard  vehicle  and  industry  standard 
backplane  bus  voltage  levels 

Uses  standard  form  factor 
and  connectors 

Select  and  implement  commercial  standard  card  and 
connector  specifications 

Meets  environmentals 

Provide  an  environment  that  adapts  vehicle 
environmentals  to  commercial  specifications 

Uses  standard 
communication  busses 

Use  popular  commercial  busses 

Follows  communications 
standards 

Use  accepted  protocols  for  communication  busses 

Comes  with  drivers  that 
bridge  component  to  OS 

Software  architecture  contains  a  “driver”  layer,  and 
implements  appropriate  standards 

Software  runtime  is 
compatible  with  OS 

Use  an  Operating  Environment  “Middleware”  to 
isolate  applications  from  OS 

Firewalls  for  hard  real  time 
applications 

Architecture  to  include  resource  management  to 
assure  resource  starvation  doesn’t  occur 
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Building  on  the  Wikipedia  21  Oct 
2008  Definition  (2  of  2) 
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Wikipedia 

Possible  Manifestation  in 

Vehicle  Design 

How  does  Vehicle  Design  Enable? 

allows 

Documentation  partitioned 

Assure  appropriate  development  documents  are 

potential 

to  describe  services, 

suitable  for  public  release 

users  to  see 

communication  &  task 

inside  all  or 

management  interfaces  of 

parts  of  the 

core  and  upgradeable 

architecture 

capabilities;  no 

without  any 

documentation  of  sensitive 

proprietary 

constraints 

capabilities 
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Do  the  Assumed  2008  Open  Architecture 
Goals  Add  New  Requirements?  (1  of  2) 
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Goals  of  Today’s  Open 
Architecture  Definition  for 

Military  Vehicles 

Possible  Manifestation  in 

Vehicle  Desiqn 

How  does  Vehicle  Desiqn 

Enable? 

Avoid  developing  unique 
designs  and  proprietary 
solutions  when  industry 
accepted  standards  exist 

Use  industry  accepted 
standards,  covered  earlier 

Covered  earlier 

Enable  use  of  commercial 
hardware  and  software 
solutions,  when  suitable 

Use  commercial  hardware 

Adapt  between  vehicle  and 
commercial  environmentals. 
Covered  earlier 

Use  commercial  software 

Provide  middleware  or 
Operating  Environment  to 
isolate  OS  from  commercial 

SW.  Covered  earlier 

Maximize  upgradeability  with 
commercial  hardware  and 
software  solutions,  when 
suitable 

Covered  above 

Covered  Above 
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Do  the  Assumed  2008  Open  Architecture 
Goals  Add  New  Requirements?  (2  of  2) 
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Goals  of  Today’s  Open 
Architecture  Definition  for 

Military  Vehicles 

Possible  Manifestation  in 

Vehicle  Design 

How  does  Vehicle  Design 

Enable? 

Support  not-well-defined 
or  constrained  increases 
in  platform  computer 
resource  needs. 

Include  reserve  space  for 
additional  hardware 

Leave  room  for  new 
chassis  or  empty  slots  in 
existing  chassis 

Include  space  for 
additional  memory 
capacity 

Include  reserve  electrical 
power  to  run  that 
hardware 

Add  reserve  to  power 
budget 

Include  present  and  future 
voltage  levels  to  power 
that  hardware 

Provide  traces  to  empty 
slots  for  possible  future 
power  supply  CCAs 

Include  space  and  data 
bus  allocations  for  user 
interface  escalation 

Provide  connector 
reserve  capacity  or  extra 
unused  connectors 
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Mapping  of  Prior  Definitions  to  U.S. 
Navy  Open  Architecture  Definition 


LvnuxWorksTM  Website 

Definition  of  U.S.  Navv  Open 

Arch. 

Covered 

Earlier? 

How  does  Vehicle  Design  Enable? 

Modular  design  and  design 
disclosure 

Yes 

Reusable  application  software 

No 

Requires  application  software  works 
with  standard  or  disclosed  APIs,  system 
unique  parameters  must  be  loaded  at 
runtime 

Interoperable  joint  warfighting 
applications  and  secure 
information  exchange 

Not 

covered, 
per  se 

Provide  computer  resources  and  APIs 
for  applications  and  a  certified 
architecture  for  secure  info,  exchange 

Life-cycle  affordability 

Not 

articulated 

Meet  all  aforementioned  requirements 

Encouraging  competition  and 
collaboration  through 
development  of  alternative 
solutions  and  sources 
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Support 

Business 

Model 


The  focus  and  funding  are  often  centered  on  delivering  tactical  systems; 
however,  a  more  holistic  focus  is  on  delivering  mission  capabilities. 
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How  do  we  move  our  focus  to  “Mission”  vs.  “System  7 


■  As  the  DoD’s  business  model  continues  to  evolve,  its  focus  on  meeting 
varying  mission  needs  within  a  bounded  O&S  budget  is  pushing  for 
some  kind  of  an  evolutionary  business  approach. 

■  Our  Processes  and  Tools  are  “System  (or  Platform )-centric” 

■  What  else  is  needed  in  order  to  perform  a  mission?  What  is  driving 
increasing  O&S  costs,  reducing  much  need  modernization  funds? 


Unfortunately,  Enabling  Systems  are  still  often  implemented  AFTER 
the  delivered  tactical  system  and  as  an  externally  designed  system.^ 
This  approach  has  been  successful  for  many  years,  but  TODAY  does 
not  lead  to  the  most  AFFORDABLE  and  mission  ready  systems. 


“Always  design  a  thing  by  considering  it  in  its  next  larger 
context  -  a  chair  in  a  room,  a  room  in  a  house,  a  house  in 
an  environment,  and  an  environment  in  a  city  plan.” 

-  Eliel  Saarinen,  Finnish  Architect 
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Sin  1 :  Insufficient  customer  needs  analysis  (i.e.  dig  deeper  and  broader) 

Sin  2:  Belief  that  all  requirements  can  be  deduced  before  the  system  is 
d  e  p  I  oy 

Sin  3:  Ignore  the  system  requirements  necessary  to  permit  enabling 
systems  success. 

Sinp4i£  Usability  design  that  is  engineering-centric  versus  user-centric. 

Sin  5:  Designs  without  the  humanlifrthe-loop  considerations. 

Sin  6:  Acquisition!  cost  focuse<||fllllllllll^ 

Sin  7:  Limited  consideration  for  net-centric  environment  integration^^ 

Approaches  for  improving  the  affordability  of  mission  success,  through 
a  more  holistic  approach  for  designing  complex  systems 
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Sin  1:  Insufficient  customer  needs  analysis 
(i.e.  dig  deeper  and  broader). 

£1  Issue:  By  not  digging  to  the  ‘root  need’,  an  incorrect 
support  enabling  system  solution  may  result. 

a  Learning  opportunities: 

■  ‘Carrier  Electro-Magnetic  Radiation  Signature’ 

■  Accurate  Design  Algorithms 

■  Successful  Design  implemented  on  a  helicopter  platform 
It  BUT:  Puts  humansjn  harms  way  (Helicopter  Pilots  sayj^  . ) 

■  Technical  merit  beauty,  but  failure  to  meet  holistic  operational 
expectations. 

■  Aerial  Common  Sensor  ISR  Army  Aircraft 

i;  Requirements  creep  to  include  cross-service  utilization  with  Navy 

■  More  customers  often  means  more  needs 

■  Tends  to  cumulatively  add  to  functionality  versus  integrate  easily  with 
existing  functionality  (a  bolt-on  functional  mentality) 

■  This  program  was  terminated  due  to  unacceptable  growth  in  weight, 
that  drove  increase  in  cooling  and  power  requirements  that  became  a 
negative  viscous  cycle. 

22  October  2008  2008  NDIA  Systems  Engineering  Conference  5 


_LI  Ml  I 


HUfiKhU  n-iav*  IS'i '-"Bill  ’.ulfi  I  ||| 


|  ig-iav*  mi'-.Tiu  i  ■ik.1. 

Sin  1  Conclusion  . . . 

■  Requirements  Management  (creep)  is  often  cited  as  a  root 
cause  for  unsuccessful  programs. 

k  Wait  a  minute!  We’re  smart  engineers,  we  know  that 
requirements  creep  is  a  problem,  So  WHY  does  it  keep 
happening?! 

■  No  schedule  time  for  sufficient  customer  needs  analysis 

■  No  holistic  enabling  system  (support)  integration 


J 


Result:  The  Primary  and  Support  Systems  are  not 
integrated  and  thus  the  requirements  evolve  separately. 


■  Thought:  What  about  the  “Development  Environment 
System?” 

Discovery  and  evolution  through  the  design  phase  is  natural. 
Needs  analysis  done  well  accelerates  functional  discovery  AND 
minimizes  unforeseen  requirements  creep. 
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Sin  2:  Belief  that  all  requirements  can  be  deduced 
before  the  system  is  deployed. 


■  As  an  extension  of  Sin  1 :  Even  excellent  needs  analysis 
may  still  assumes  a-priori  knowledge  of  the  full  breadth  of 
the  operational  uses,  environments,  laws,  etc.  This  is 
typically  NOT  a  reasonable  assumption. 


■  Learning  oppoguiflities^^^^^^^^^^^^^^^^^H 

■  Acoustic  Rapid  COTS  Insertion  (ARCI)  Navy  program 

■  Requirements  are  the  variable;  cost  and  schedule  are  locked 

■  System  obsolescence  (support)  and  functional  growth  are  merged 
i|§  The  system  to  evolves  capabilities  annually 


■  Aircraft  weight  grows  at  the  rate  of  ‘1  -pound  per  day  of 
deployed  operations’ 

■  Bolt-on  functionality  growth  approach 

■  ~  300  pounds  per  year  for  20  years  06000  pounds! 

I  Also  additions  to  size,  weight,  power,  cooling,  logistics  footprint 

■  Knowing  this  military  history,  do  we  design  for  this  in  mind?? 
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Sin  2  Conclusion  . . . 
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■  Clearly  define  a  tradable  space  for  system  evolution  that 
becomes  your  decision  algorithm  for  changes, 

■  Consider  Life  Cycle  Cost,  Reliability,  Risk,  and  Performance  as  a  4- 
dimensional  trade  space  as  a  means  of  managing  growth  requests. 

II  Ensure  |haf  the  architecture  is  truly  open  and  permits 
evolutior§  of  the  underlying  hardware  and  software  physical 
solutions 


Army  OODA  Loop  -  In  battle  situations  there  is  a  constant  loop  of 
Observe,  Orient,  Decide  and  Act.  Continuously  manage  emergent 
information. 


Suggestion:  Instead  of  fearing  requirements  creep,  we  should 
embrace  the  dynamic  nature  of  a  system  design  through 
incremental,  spiral,  and  agile  development  methods. 
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Sin  3:  Ignore  the  system  requirements  necessary 
to  permit  enabling  systems  success. 

■  Issue:  Enabling  systems  often  do  not  get  ‘equal  design 
focus’  and  yet  the  impacts  of  a  Jaw  in  the  enabling  systems 
are  oftei|  program  show-stoppers 

H  Learning  Opportunities:  _ 

■  F-1  I  ^Nighthawk  -  First  stealth  fighter 

■  Disruptive  technology  that  revolutionized  battle  options 

■  Still  one  of  the  finest,  most  technologically-advanced  fighters  in  aviation  history. 

■  World-class  mission  capabilities  as  evidenced  during  Desert  Storm 

■  The  initial  design  focus  was  stealth  fighting  capability,  quickly. 

■  The  enabling  system  operational  consideration  was  given  a  “back-seat”. 

■  The  enabling  systems  also  have  world-class  records  with  keeping  the  aircraft 
flying;  however,  the  costs  for  this  support  are  quite  high.  . 

■  New  environments  (Desert  to  Rain  forest  to  South  Pole),  New  uses 
(unforeseen  requirements),  and  Emerging  threats 

■  These  conditions  can  take  an  apparently  successful  system  solution  and  render  it 
unsuccessful.  Desert  Storm  was  an  eye-opener  for  the  assumption  that 
performance  and  reliability  were  the  same  in  a  high-grit,  high-heat  environment. 

■  Getting  the  enabling  system  materiel  Ip  country”  was  efficient,  but  made  useless 
because  the  enabling  system  was  not  designed  to  get  them  to  “point  of  use” 
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■  Does  your  system  requirements  management  database 
have  derived  supportability  requirements  included? 

■  Typically  foot;  howeve|,  the  high level  suppodability 
Requirements  are  often,  delineated  ill  thje  Originating 

Requirements  Document  or  the  Statement  of  Work. 


The  “best  performing  system  design  ever”  will  still  fail  if  the 
consumables  and  logistics  tail  are  not  sufficient  to  ensure  system 
Operational  Effectiveness. 


Integrated  design  for  support  -  Supportable  design  - 
Support  the  design  affordably 
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Sin  4:  Usability  design  that  is  engineering-centric 
versus  user-centric. 
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■  Issue:  Designers  are  too  often  enamored  with  functional 
elegance  arad  flexibility  making  everything  ijaghe  solution 
a  variable;  however  this  demands  too  much  user 
interaction  and  intimate  process  knowledge  in  order  to 
properly  provide  inputs  a|d  interpret  system  outputs. 


J 


Learning  Opportunffles: 

■  MOP4  operations,  where  soldiers  are  wearing  Chem-Bio  suits 

■  Allow  for  system  operations  with  bulky  gloves 
ii  Extreme  environments,  fatigue,  heat,  cold,  etc, 

■  Move  toward  Autonomic  Logistics  versus  traditional  support 
options 

^iMore  system  integration 

■  User-centric  designs  and  focus,  versus  functional" 'decompositions 
i  Learning  systems  versus  static  systems 
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Conclusion  4  . . . 
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.  Marine  Corp  Embedded  Platform  Logistics  System 
(EPLS)  Gene  Morin,  Program  manager  is  quoted  as 
sayiflg,  “I  want  my  Marines  to  have  their  fingers  on 
triggers  and  not  on  keyboards.” 

•  This  says  it  concisely.  Logistics  support  systems  should 
“make  things  happen,  when  they  need  to  happen,  and 
wijhoif  humalinterveliolif  at  all  possible”. 


Suggestion:  Carefully  trade  functional  flexibility  with  user 
simplicity.  To  this  end,  possibly  consider  multiple  modes,  Users 
and  Use  Cases.  WHEN  at  a  minimum?  At  all  Design  Reviews. 


User-centric  design  methodologies,  and  using  the  recommendation 
from  Sin  1,  ensure  that  deep  dive  analysis  distills  the  relevant 
information  which  can  be  absorbed  in  a  “User  glimpse”. 
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Sin  5:  Designs  without  the  human-in-the-loop 
considerations. 
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■  Issue:  Major  Total  Ownership  Cost  (TOC)  and  System 
effectiveness  are  driven  by  Humans  in  the  System 

■  Learning  Opportunities: 

■  Air  Bag  Design  (initial  designs) 

■  Requirements  defined  in  early  ’80’s  and  deployed  in  early  ’90’s 

■  Design  for  men  only  (50  Percentile  Male) 

■  The  Air  Bags  themselves  were  causing  female  fatalities 

■  U2  Spy  Plane* 

■  Disruptive  technology  with  landmark  capabilities 
||  70, 000+  ft  altitude  and  extended  loiter  times* 

H  Requires  space  suits  for  pilots,  no  relief,  no  physical  movement 
possible,  etc.  The  pilot  was  a  back-seat  consideration. 

Would  we  design  the  U2  today  or  might  we  use  a  UAV  for  the  mission? 

*  Source:  www.wikipedia.com 
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Conclusion  5  . . . 


j_u  i 


■.-.iimu  n-ii'Uti  rwviiiu  <v4u  i  m; 


5 


■  Fight  the  tradition  of  ‘the  way  things  have  always  been 
done’,  and  intentionally  put  the  human  inside  your 
system  desigrl  requirements  boundary. 

■  It  may  then  become  obvious  that  |he  human  is  being 
expected  to  do  ‘too  much’,  and  therefore,  the  desigiH 
team  should  explore  automated  arid  autonomous 
alternatives  for  the  system  sol 


How  does  the  human  interact  with  your  system? 

Human  at  risk?  Human  overloaded  (information,  attention,  actions)? 

You  may  also  discover  that  the  maintenance  and  upgrade  for 
Automation  is  much  cheaper  than  the  humans  that  are  freed  up. 
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Sin  6:  Acquisition  cost  focused. 

■  Issue:  A  focus  on  acquisition  cost  alone  when  making  design  decisions 
is  a  typical  approach;  however,  leaves  much  affordability  opportunities 
unleveraged. 

■  70%  of  the  O&S  costs  are  determined  as  soon  as  requirements  are  set. 

■  Wait  a  minute  |.  .  Isn’t  this  a  good  thing? 

■  Learning  Opportunities;  ^  __ 

■  F-35  Multi-national  and  Joint-forces  Fighter! 

■  Mission  Reliability  (Operational  Availability)  is  a  Key  Performance  Parameter 
(TOC  too) 

■  KPP’s  for:  Sortie  Generation  Rate  and  Logistics  Footprint 

■  $135  B  or  a  56%  estimated  TOC  savings  compared  to  legacy  systems 

■  Mission  Reliability  of  over  90%  and  a  30  day  self  sustained  mission 

■  12%  or  $16B  is  expected  to  come  from  Enabling  System  Automation  Prognostics 
&  Autonomies 

I  Advanced  Amphibious  Assault  Vehicle: 

■  Acquisition  program  focused  on  mature  technology  and  O&S  costs. 

■  Program  office  co-located  with  contractor  facility  and  extensive  use  of  end  user 
assessment  of  system  operational  effectiveness. 

■  Extensive  reliability  testing  with  common  components  of  other  Marine  Corp. 
weapons  systems. 

■  Initial  O&S  cost  savings  =  $29  million. 
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Conclusion  6  . . . 
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■  Evolutionary  Acquisition  approach  to  design. 

■  Incremental  development  which  assumes  life  Cycle  as  a 
requirement  of  the  Enabling  System  AND  also  as  a  Mission 
Requirement  of  the  Primary  System 

■  The  desigti  “endi  state’meeds  |o  include  |he  system  life 
cycle  through  to  disposal. 


TOC  and  Performance  must  be  of  equal  importance  in  the  design 
trade  space. 


Move  is  from  a  ‘point  solution’  (Performance)  to  an  ‘evolutionary 
solution’  perspective. 
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Sin  7:  Limited  consideration  for  net-centric 
environment  integration. 
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■j  Issue:  In  a  typical  development  environment,  the  design 
_Jeam  focuses  on  the  requiremeiatslbr  which  they  are  paid 
to  ianovatively  solve.  Also  typically,  this  is  viewing  the 
system  as  a  starld-alone  entity  with  interfaces  to  the 
world  around  it. 


The  challenge  is  not  as  simple  as  ‘does  this  system  talk 
to  that  system’  but  rather  the  emergent  system  of 
systems  capabilities  add  challenges  that  occur  when 
systems  are  corrected  wilhiiya  network-centric 
eivirodmeifl 


22  October  2008 


2008  NDIA  Systems  Engineering  Conference 


17 


Conclusion  7 
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For  all  I^J  AND 
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•  pncs"  The  total  performance  of  the  Network-Centric  Syetem 

•  P,  ■  The  performance  capability  of  a  Stand  Alone  Syetem  (no  network  connection) 

•  Pj  ■  The  performance  capability  of  a  Stand  Alone  Syetem  (no  network  connection) 

•  N  ■  The  number  of  Independent  Syeteme  (Network  Nodee) 

•  M  ■  The  number  of  Independent  functional  connection  pathe  for  a  P,  and  Pj  pairing 

•  P,  fl  Pj  ■  Thle  Intersection  represente  the  resultant  performance  from  the  system 
connectivity,  whloh  oould  be  Zero  If  there  le  no  eyetem  advantage  or  detractor, 
Positive  If  the  connectivity  advantages  the  ConOps  (Mission  Needs)  or  Negative  If  the 
connectivity  Is  not  required  by  the  ConOps  (l.e.  outside  of  the  mission  performance 
boundaries) 


The  net-centric  whole  is  greater  than  the  sum  of  the  systems. 
Pairwise  additive  capabilities,  Triples  additive  capabilities, . . . 

Some  capabilities  are  Good  and  some  are  Negative  in  YOUR  system. 
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Principal  Recommendations 
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■  Focus  on  the  mission  needs,  make  time  for  and  dig  deep 
to  root  out  the  true  stakeholder  needs. 


Ensure  the  system  is  affordably  evolvablejthroughythe 
support  life  cycle. 

The  Primary  and  Enabling  Systems  must  be  holistically 
designed  as  a  single  complex  System  of  Systems. 

Document  and  decompose  ALL  of  the  requirements. 


B  A  combined  and  equafl  focus  ofl  pejformafiice  and  suppoif 
requirements  during  design  for  total  system  performance 
responsibly. 


Thought:  These  recommendations  outline  a  more  inductive 
approach  to  our  traditionally  deductive  engineering  paradigm. 
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US  Joint  Forces  Command 
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Current  S&RL  Global  Governmen 
Participants/CRADA  Holders/ 


■  OSD  -  ATL  (Acquisition  Technology  &  Logistics) 

■  DISA  (Defense  Information  Services  Agency) 

■  JFCOM  (Joint  Forces  Command) 

■  NNWC  (Naval  Network  Warfare  Command) 

■  MARCORSYSCOM  (  Marine  Corps  Systems  Command) 

■  NATO  (North  Atlantic  Treaty  Organization) 

■  EDA  (  European  Defense  Agency) 

■  ACT  (Allied  Command  Transformation) 

-  NC3A  (NATO  C3  Architecture) 

■  DAU  (Defense  Acquisition  University) 

■  ONR  (Office  of  Naval  Research) 

■  DLA  (Defense  Logistics  Agency) 

■  BTO  (Business  Transformation  Office) 

■  Force  Transformation  Office  (Sense  &  Respond  Logistics) 

■  DOD  Australia 


Discussion  Objectives 
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■  Show  the  Strategic  Plan  to  develop  a  Global  Network 
Centric  Logistics  Environment. 

■  Introduce  Network  Centric  Engineering  and  its 
application  on  various  projects. 

■  Designing  a  Logistics  NCE  using  Operational 
Descriptions,  Standards,  Patterns,  and  Building  Blocks. 

-  Requirements  Validation 

•  Operational  Descriptions;  SCOPE;  Well  Formed  Requirement 

-  Standards 

-  Patterns 

-  Building  Blocks 


Network  Centric  Logistics 
Strategic  Plan 


1 .  Identify  &  Enhance  Network  Centric 
Logistics  Requirements,  Standards, 
Patterns,  and  Building  Blocks. 

2.  Build  on  this  framework  for  a 
global,  commercial  &  government, 
logistics  community  of  interest 
focused  on  collaboration. 

3.  Apply  the  processes  &  toolset  to 
integrate  global  network  super 
nodes. 

A.  SCLA/DOD:  JDDSP  (Joint  Power 
Projection  Support  Platform) 

B.  US  DOD/NATO/AUSCANNZUKUS: 
Joint  &  Coalition  SeaBase 

C.  NATO:  NRF  TC  (NATO  Response 
Force  Training  Center) 

D.  Commercial  Global  Logistic 
Distribution  Centers 


“Just  in  Time  Delivery”  to  the  Military,  Using  Commercial 
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Network  Centric  Engineering 
Core  Competencies 


Requirements  Capture 

-  Operational  Description 

-  CONOPS 


-  JCIDS  Processes  and  Documents 

-  SCOPE  (Systems-Capabilities-Operations-  Programs-  Enterprises)  Analysis 

-  WFR  (Well  Formed  Requirement)  Model 

-  Business  Process  Mapping 

-  Other  Tools  (  SCOR,  NCAT,  etc.) 

■  Architecture  and  Lexicon  Development 

■  Modeling  and  Simulation 

■  Standards  Framework  Design  and  Development 

-  Data  Sharing  Concept  and  Design 

■  Operational  and  Technology  Capability  Patterns  and  Guidance 

■  System  and  Network  Selection  from  the  Building  Blocks  Repository 

■  Prototype  Building 

■  Test  and  Experimentation  (Build  a  little,  Test  a  little,  Learn  a  Lot) 

-  Human  Systems  Integration  (DOTMLPF) 


Network  Centric  Engineering 
for  JDDSP  Example 


■  Requirements  Capture  (Business  Process  Analysis) 

-  CONOPS  =>  Initial  Capabilities  Documents 

■  Process  Mapping  &  Modeling  Operations  Activities 

-  SYSML  and  Other  Models  for  Various  Use  Cases  and  Scenarios 

■  Architecture  Design  &  Development  (Service  Oriented  Architecture  Artifacts) 

-  Standards  Selection-Integration  (NSWG,  DISR,  SCOPE  Analysis,  PFC,  ...) 

-  Service  Oriented  Architecture:  GIG  Integrated,  Open  Standards,  XML,  ... 

■  Site  Physical  and  Cyber  Site  Security  Plan  (Information  Assurance) 

■  JDDSP  Experimentation  Plan  Development  (Operational  Test-bed  Activity) 

-  Pacific  Northwest  Corridor  (Force  Deployment)  Experiment 

-  Dole  Pacific  Shipping  (Commercial  Distribution)  Experiment 

-  TATRC  Class  VIII  (Force  Sustainment  -  Sense  &  Respond  Logistics)  Experiment 

■  Sea-Basing  Template  (JDDSP  Interface) 

■  Prototype  Build  (System  of  Systems  Integration) 

■  Execute  Experiment  to  Fill  Gaps  in  Rationale 

■  Perform  Demonstration 

-  Human  System  Integration:  DOTMLPF 

-  Mission  Capability  Packages 
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5.  SCLA-JDDSP  data  exchange;  access  to  Web 
Services;  exchange  of  signed  and  secured 
documents. 


Data  exchange  over  secured  channel 
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6.  Deployment 

7.  Support 

8.  Upgrade  or  Disposal 


Supports 
Layered 
Quality 
of  Service 

Network 

Centric 

Analysis 

Tool 

(NCAT™) 

Building 

Blocks 

(BB) 


Expeditionary  Force  Deplo 
Operational  Description 


w 


■  NCOIC  -  ONR  JDDSP  -  SR  LLC  Subject  Matter  Experts 
develop  an  Expeditionary  Force  Deployment  Operational 
Description  (EFD  OD),  Mission  Threads,  Scenarios,  and 
CONOPS. 

■  EFD  OD  informs  the  list  of  standards  and  application 
processes  on  information  security  and  other  functions  for  IT 
network  design. 

-  Defined  the  Potential  Patterns  to  Define  Log  Domain. 

-  Initiated  Building  Blocks  Database  for  Log  Domain. 

■  Develop  NIF  patterns  that  describe  Interoperability  Criteria 
to  accomplish  the  Logistics  “Total  Asset  Visibility”  mission  and 
use  existing  commercial  infrastructure  to  deploy/sustain, 
without  disrupting  commercial  enterprise. 

-  e2e  visibility  replaces  30-day  “Iron-Mountain”. 

-  Logistics  UDOP  picture  provides  max  collaboration. 


“Just  in  Time  Delivery”  to  the  Military,  Using  Commercial 
Transport  Mechanisms  (Wal-Mart  and  FedEx  style  delivery) 


TCAIMS-II 


ICODES 


GTN 


GATES/ 

WPG 


AALPS  ^ 


Others 


Infrastructure 

Components 


Others 


Autonomic 

Logistics 


Net-Centric 

Data 

Environment 


Enterprise 

Services 

Management 


f  \ 

Transportation 

Management 

V _ _ J 

m  In-Transit 

1  Visibility 

LOGCOP 


Inventory 


RFID 

Locator 


DTCI 


IRRIS 


Others 


SenseResponder  LLC  assists  in  Requirements  Validation;  Standards  Identification 
and  Cross  Linking;  Pattern  Development  for  Interoperability  Guidance;  and  a 
COTS/GOTS  Products  &  Services  “Building  Blocks”  Repository. 


Recommends 


Designing  an  NCE  (Network 
Centric  Environment) 


Informs 


NCAT 


Guides 


:  Requirements  Validati 


■  OD  (Operational  Description) 

■  SCOPE  (Systems  -  Capabilities  -  Operations  - 
Programs  -  Enterprises)  Analysis 

■  WFR  ( Well  Formed  Requirement)  Model 


Logistics  “Operational”  Capability 


End-to-end 

communications 


Total  asset  visibility 


Rapid  delivery  of 
mission-ready  forces 


Reduced  inventory, 
smaller  footprint, 
faster  response 


Rapid  distribution  of 
tailored  support  packages 


Logistics 

decision 

superiority 


Information  fusion 


Bottom 

line: 


Forces  in  theater  —  whether  forward-stationed  or 
deployed  —  deliver  more  capability,  require  less  support 


>* 


Logistics  “Technical” 


rroauci  uaia 


Contract 

Requirements 


Weapon 

System 

Requirements 


DESIGN 


Key 

Supplier/Prime 

Requirements 


As 
|  Designe 


As 


'roducts  itfaintainpd 


Support  System 
Integrator 


Tech 

Data 

Training 

SE 

LSA 

Analysis 

Financial 

Budgets 

Congress 

QDR 

DoD/OSD 

_ Policy _ 

^Translation  Data  Standards 


PRE  POST 
DELIVERY 


if) 

+-» 

O 

3 

~U 

O 

L- 
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a 

3 

CO 

DD250 

USA 


USN  -■<$>-■ 


USAF 


Service 

Data 

Warehouses 


Tech  Data 


Training 


Support 

Equipment 


Weapon  System 
End  Item 


Line 

Maintenance 

Supply 

Management 

Depot 

Maintenance 


DO-XX 


CAMS 


NALCOMIS 


TAMS 


U.S.  Government 


Operating  and  Support  Commands 


Operations 

And  Maintenance  Data 


Logistics  Architecture  Solution 


Users 

Presentation 

Applications 


Standards- 

Based 

Integration 


Common  Operational  Picture 


•  •  • 


Role-based  Single  Sign-on 


Enterprise  Portal,  Collaboration,  Business  Intelligence,  Wireless 

Resource 

Management 

QASP 

Compliance 

Program 

Management 

Knowledge 

Management 

Multi-Level  Label  Security 


Program 
Info  Sharing 
Partners 


Central  Repositoi 


Multi-Level  Label  Security 


Data.Warehouse 


Connected 

Legacy 

Systems 


Finance 


PDM 


SCM 


HR 


CAD 


•  •  • 


Data  Mining 


Sample  Sense  &  Respond  Logistics 
Operational  Description  Document^ 


1.  Introduction 

2.  Architecture  Principles  and  Artifacts 

3.  S&RL  Problem  Description 

4.  S&RL  Interoperability  Solutions 

5.  Attributes  or  Global  Aspects 

6.  Enabling  Technology  Patterns 

7.  Interoperability 

8.  S&RL  Open  Protocols  and  Standards 

9.  Business  Model  Implications 

10.  Applicable  NCOIC  PFCs  and  External  References 

1 1 .  Network  Centric  Engineering  the  JDDSP  (Joint  Deployment 
Distribution  Support  Platform) 


Well  Formed  Requireme 


i 


CUSTOMERS 

Well  Formed  Requirements 


1 
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Convergence 

Collaboration 


INDUSTRY 

Standards,  Patterns  and  Building  Blocks 
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Coherence 

Conformance 

Consensus 

Convergence 

Collaboration 


STDS 


STDS 


fli 
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STDS 
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STDS 

STDS 

_ / 

Two  Sides  of  the  Same  Coin 


Dimensions  of  a  Require 


■  Function 

-  what  is  to  be  done 

-  Usually  text  description  today,  but  could  be  a  video,  simulation,  animation,  etc. 

-  Granularity  can  be  from  a  capability  to  a  service 

■  Constraints  -what  tolerances  must  be  met 

-  Measures  of  Effectiveness  (MOE) 

-  Measures  of  Performance  (MOP) 

-  Measures  of  Net-Centricity  (MON)  -  new  and  analyzed  in  NCOIC  SCOPE  model 

-  Measures  of  Satisfaction  (MOS)  -  new  to  DoD 

-  Size,  Weight,  And  Power  (SWAP) 

-  Costs  and  Schedules 

-  Risk  Tolerance  (TRL  -  Technology  Readiness  Level) 

-  Miscellaneous  (a.k.a.  the  “ilities” 

■  Operational  Context 

-  Physical  Environment 


From  Requirements  to 


■  Function  and  Operational  Context  are  usually  well  understood  and 
unchangeable  [^without  doctrine  or  CONOPs  rework] 

■  Solution  usually  requires  trade-offs  among  the  multiple  constraint 
dimensions 

-  For  example  trading  reduced  durability  for  lighter  weight 

Some  constraints  are  more  inflexible  than  others  or  have  tighter  range  of 
values  in  different  Operational  Contexts 

-  Reliability  (MTBF)  for  space-based  radio  transmitter  on  a  missile  launch  early 
detection  satellite  much  higher  and  less  negotiable  than  for  a  tower-based  radio 
transmitter  for  the  Voice  of  America 

"  Selected  solution  is  often  the  alternative  that: 

-  performs  the  function... 

-  in  the  operational  context... 

-  and  “best  fits”  the  customer  and  contractor  “agreed  upon”  blend  of  constraints 
resulting  from  trade-offs  determined  during  architecture  or  system  design 


Policy  vs.  Contractual  vs 
Service  Level  Agreemen 


■  For  a  given  Function  In  a  given  Operational  Context: 

-  Some  requirement  dimensions  will  be  best  specified  as 
contractual  obligations  such  as  acceptance  criteria  or 
incentive  fee  items 

-  One  time  measurement  against  specification 

■  Some  requirement  dimensions  will  be  best  specified  as 
Service  Level  Agreements  (SLAs) 

-  Continuous  measurement  against  specification 

■  Some  requirement  dimensions  will  be  consensus 
globally,  some  nationally,  some  military  vs. 
commercial,  and  some  within  COI 


Well  Formed  Requirement  -  Kiviat  Chart 


MOE  and  MOP  in  this  example  are 
from  Integrated  Broadcast  Service 
(IBS)  Sources  Sought  RFI  from 
USAF  ESC  March  20,  2006  and  are 
provided  strictly  as  an  example. 


11.2  Million  Messages/day 

Accept/Deliver  Data  in  8  Legacy  Formats 

Concurrently  accept  CMF  reports  via  S&N  RF 
Networks,  and  Telephony 


User,  Regional,  COCOM  set  priorities 


Measures  < 
Performance 
(MOP) 


500  Simultaneous 
User  Profiles 


End-to-End 
Response  Time 


Miscellaneous 
(a.k.a.  -  ilities) 


Testability 
Maintainability 


Durability 


Space,  Undersea, 
Airborne 


Others:  temperature,  humidity,  press 
dust,  fungus,  Electronic  Attack  (EA), 


Deployment 

Schedule 


Cost  &  Schedule 


Technical  Readiness  Level  (TRL) 


Relating  Systems,  Capabilities,  Ope 
Programs,  and  Enterprises  (SCOBB 


“Intergalactic  Radiator” 
by  Capt  Yurchak 
For  SCOPE  illustration  only 


Tactical  C2  MCP 


ISR  MCP 


Navigation  MCP 

Individual 
Programs/Systems 


or  System  of  Systems 
Missile  Defense  MCP 


Time-Critical^Strike  MCP 


Enterprise 


Current  Navy  Warfare  Sponsors 
EXW  SUW  USW  AW 
N75  N76  N77  N78 


Budgets  allocated  vertically 


ASW 

N74 


Systems  of  systems  often  . 

aligned  to  these  capabilities  Operaf/cWs 


[ often  in  and,  out  ofjjage ) 


Illustrates  Complex  Dependencies  in  Capability  Acquisition 


ii 


The  Role  and  Value 
of  the  SCOPE  Model 


SCOPE.  Systems,  Capabilities,  Operations,  Programs,  and  Enterprises 


High  Level  Models 


Enterprise 

Models 


Models  of  The  SCOPE  Model 
Customer  measures  needs  of  each 
Objectives  domain  in  many 

dimensions... 

Net-Centric 
Interoperability 


Domain 

General 

Architectures 


Specific  Node 
Architectures 

Military 

Implementation 


Net  Awareness 


Autonomy 


Tailored  Interoperable? 


Service  Orientation 


E-commerce 
Web  Services 


Capability 
Scope 


...  and  each  domain 
often  has  different 
needs,  characterized 
via  the  SCOPE  Model 


Government/ 

Commercial 

Implementation 


The  Role  and  Value 
of  the  SCOPE  Model 


High  Level  Models 


Enterprise 

Models 

Models  of 
Customer 
Objectives 

Net-Centric 

Interoperability 


Domain 

General 

Architectures 


Military 


Specific  Node 
Architectures 


Military 

Implementation 


Through  SCOPE  Model  analysis, 
the  aspects  of  interoperability  across 
dissimilar  nodes  can  be  assessed 


Tailored 


Net  Centric 


Tailored  lnteIyPer^jl'ty 


E-commerce 
Web  Services 


Civil/ 

Commercial 


> 


Government/ 

Commercial 

Implementation 


Capability  Scope  Dimensions 


DimensiorT'\^ 

Narrower  Scope  Broader  Scope 

Overall  Scope  and 
Types  of  Enterprise 

Single  Unit 

Single  Service  or 
Agency 

DoD-Wide 

World-Wide 

Capability  Breadth 

Single  Functional 
Domain/Service 

Multi-Domain,  Multi- 
Service 

Multi-Dept,  NGO, 
Industry 

Coalition,  Multi- 
Enterprise  Type 

Capability  Depth 

Single  Level 

Two  Levels 

Three  Echelons 

Four  or  More 

Echelons 

Organizational 

Model  and  Culture 

Rigid  Hierarchy, 
Vertically  Integrated 

Adaptive  Hierarchy, 
Interact  Horizontally 

Flat,  Empowered, 
Open  to  Partnering 

Adaptive,  Social, 
Interdependent 

Unity  of  Life  Cycle 
Control/Alignment 

Single  DoD  Acquis. 
Exec 

Multiple  DoD 

Acquis.  Exec 

DoD  &  US  Syst. 
Owners 

Multi-National  Syst. 
Owners 

Acquisition 
Congruence  (SD) 

All  Systems  on  Same 
Timeline 

Timeline  within  2 
years 

Timeline  within  5 
years 

Timelines  >5  years 
apart 

Semantic 

Interoperability 

Single  Domain 
Vocabulary 

Multi-Domain 

Vocabulary 

Single  Language 

Multiple  Languages 

Operational 

Context  (SD) 

Single  Ops  Context 

Multiple  Ops 

Contexts 

Future/Past 

Integration 

Hypothetical 

Entities 

■  Identification 

■  Analysis 

■  Linked  to  Architecture  Role,  Products 
Guidance 


Linking  Network  Centric  Guidance  a 
Technology  with  Standards 


BB  Repository 


Standards 


H  Productsr 


Standards  supported 
By  product 


ZL 


L  Arch  Role*- 


Architectural  Role  supported 
By  product 


Arch 


Use'oLproduct  in  NCP 
Identified  architectural  roles 


Standard: 


NCOIC 
Extended 
ISO  ICS  &  FEA 
Classification 
Model 


Use  of  standard  in  NCP 
Identified  architectural  roles 


Arch  Role 


Standards  Structure 
Guidance 


Standards 
Semantic 
Classification 
(SSC)  NC p. 


Network  Centric  Patterns  (NCP) 


Direct  Product  Mapping  of  Standard 
Product  Categories  and  Products^ 


Federal  Enterprise  Architecture 


Product  Categories 


Benefits  of  Standards 
Classification 


■  Aggregation  of  knowledge  by  the  international  community  about  the 
architectural  uses  of  standards  for  Network  Centric  Operations. 

■  Enables  any  organization  to  contribute  to  and  discover  architectural  uses  of 
standards. 

■  Evolution  of  a  standards  framework  about  concepts  of  architectural  roles,  a 
vocabulary  to  label  them,  and  a  model  to  relate  them. 

■  Enables  Product  Managers  to  determine  if  their  products  support  the  NCP 
standards  guidance  and  discover  International  uses  of  standards  for  the 
architectural  roles  of  their  products. 

■  Enables  your  organizations  products  and  services  with  standards  applications 
to  be  integrated  into  Federal  Enterprise  Architecture  reference  models  and 
thereby  the  architectural  and  implementation  plans  of  organizations  complying 
with  the  FEA. 


Architectural  Role/Technology 
Classification  Model 


Application  Domain 

1 

Mission 

- ► 

Network  Operations 

- 1 - 

i 

Policy  Languages 

Lifecycle 


Intelligence 


Command 


Human  Interface 


Supply  Chain 


Products 


Emergency 

Response 


Aviation 


Military 

Coalition 


Visual  Representation 


Models 


Maps 


Symbols 


Reference 

Models 


Architecture 


System 


Performance 


Tt 


Naming 


UID 


Collaboration 

=1 - 


Process/Activity 


Codification 


Network  Management 


Technology  Network 


Interface  Profiles 


Communications 


Service  Description 


Service  Access 


Service  Composition 


Information/Data 


H 


COI  Vocabularies 


Domain  Schemas 


Semantic 

Representation 


Radio 


Domain  Ontologies 


Markllp 


Protocols 


Exchange  Data 


Content  Type 


Message 


Metadata 

Sensors 

- 1 - 

Services 

Security 


GeoSpatial 


L 


Location 


Standard 

Context 


Individual  or 
Profile/Pattern 


Level  of 
Internationality 


QOS 


Measurement 


Web  Content 


Availability/Reliability 


Multimedia 


AAP  Standards  Framework  for 
Logistics  Domain 
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•  Architectural 
Guidance 

•  Standards 
Guidance 


Automatic  Identification  and 
Data  Capture  (AIDC) 


Optical  Character  Recognition 
(OCR) 


IP-MTOPS 
Web  Interface 


802.11  Network 
Presence  Tracking 


1  M 

Cell  Phone  Tracking 

Smart  Cards 

Biometrics 


Barcodes 


SM21  Tracking 
Experimentation 
Database 


□  □  □  □ 


□  □  □  □ 


Voice  Recognition 


RFID 

Fixed  Readers 

Trade  Corridor 
Operating  System 


Note:  This  is  an  illustrative  concept  diagram.  Firewalls  and  other  details  are  omitted  from  the  depiction 


■  Net-Centric  Pattern  Technology 

■  Specialized  Frameworks 

-  Information,  Communications,  Services,  Security 

■  Interoperability  Criteria  and  Guidance 

-  Building  Codes 


Three  Major  Categories  of 
NCOIC  Patterns  A 


OPERATIONAL 
DOMAIN  “A” 


OPERATIONAL 
DOMAIN  “B” 


Composite  Pattern 


M  ^ 

W  M  \  Vri 

A  ^ 

CAPABILITY 

CAPABILITY 

CAPABILITY 

CAPABILITY 

PATTERN  1 

PATTERN  2 

PATTERN  3 

PATTERN  4 

Net-Centric  Total  Asset  Visibility  Com 
Pattern  and  Component  Capability 


Asset  Information 
Sharing  Pattern 


Chain  of 
Custody 
Pattern 


Chain  of  Custody  Pattern  - 


Net-Centric  Pattern 
Guidance 


xx 


Net-Centric  Process 
Element 


Net-Centric 

Product 

Categories 


Pattern  Guidance 
Mapped  to 
Product  Categories 


Identifies  net-centric 
Capabilities  for  each 
scenario 


10 
BB 

Repository 


BB  Mapped 
To  Patterns  & 
Products 


Pattern  Guidance 
Mapped  to 
BB  Products 


Identifies 

Op  &  Tech  Problems 
for  each  capability 


Set  of 

Op  &  Tech  Mitigation 
Guidance 


Set  of 

Op  and  Tech  Standards 
Guidance 


Set  of 
Op  &  Tech 
Conformance 
Criteria 


Net-Centric 

Net-Centric 

Problems 

Pattern^ 

Net-Centric 

Guidance 

Net-Centric 

Pattern 

Scenario 

Model 

Standards 

Conformance 

Capability 

Elements 

Guidance 

Questions 

Net-Centric 

Operational 

Capability 


et-Centric 
Operational 
Problems 


Net-Centric 

Technical 

Capability 


Problem  Mitigation 
Guidance 


Net-Centri 

Operational 

Guidance 

Model 


et-Centric 

Technical 

Problems 


Problem  Mitigation 
Guidance 


Net-Centric 

Operational 

Standards 

Guidance 


Net-Centri. 

Technical 

Guidance 

Model 


Net-Centrtf  7  b 
Operationa 
Standards 
Compliance 


Net-Cent 
Technical 
Standards 
Guidance 


Net-Centric 

Technical 

Standards 

Compliance 


7d 


Network  Centric  Logistics  Environment 
Product  Categories  (P1-P10)  ^ 


Net-Centric 

Logistics 

Terminal 


Global  Net-Centric 
Logistics  Capabilities 


>■ 


> 


> 


> 


P10.  Net-Centric 
Logistics 
Gateway 
(interfaces 
Local 

Logistics  system 
with 

Common  Logistics 
global 

Capabilities) 


< 


< 


< 


< 


Net-Centric 
Logistics  Gateway 


> 


> 


> 


> 


- -A 

P5.  Net-Centric  Local 
Asset  Allocation 
Planning  System 
(Adapter) 

P6.  Net-Centric 
Local  Asset  Allocation 
Planning  Services 
(Adapter) 


P7.  Net-Centric  Local 
Asset  Allocation  Planning 
Information  Management 
System 
(Adapter) 


P8.  Net-Centric  Local 
Sense  &  Response 
Notification  &  Information 
Sharing  & 

Syndication  System 
(Adapter) 


Native 

Local 

System 


Net-Centric  Logistics  Non 

System  Adapters  ▼  Net-Centric 

Logistics  System 


Framework  to  Pattern  to 
Guidance  Matrix 


AAP  Standards  Framework 

Element 

Role  in  AAP  Pattern 

Standards  Guidance 

Supply  Chain  Model 

The  SOCR  model  identifies  typical  supply  chain  AAP  business 

Supply  Chain  Council  -  SCOR  Supply  Chain  Operations 

level  processes  and  activities  defined  which  are  then 
supported  by  the  processes  and  activities  in  the  BPMN 
model. 

Reference  model  PI  and  EP1  Planning  Operations 

AAP  collaborative  Net-Centric  Model 

This  business  process  model  describes  the  net-centric  interactions 

OMG  -  BPMN  Business  Process  Model  Notation 

across  a  set  of  business  activities  for  multiple  organizations 
participating  in  a  joint  asset  and  logistics  planning 
operation.  The  model  is  specified  in  BPMN  standard 
notation  and  is  exchangeable  across  BPMN  tools  using  the 
WfmC  XPDL  standard.  The  top  level  coordination  planning 
messages  associated  with  synchronized  business  process 
activities  are  defined  in  the  AAP  BPMN  model  as  well  as 
the  relevant  scoep  ofteh  passed  data  objects. 

WFMC  -  XPDL  XML  Process  Definition  Language 

AAP  Net  Centric  Planning  BPEL 

This  set  of  BPEL  processes  are  derived  from  the  AAP  BPMN 

OASIS  -  WS-BPEL  Web  Services  Business  Process 

model  and  control  the  orchestration  of  AAP  Planning 
services. 

Execution  Language 

AAP  Planning  Services 

This  is  a  set  of  common  planning  services  that  enable 

W3C  WSDL 

collaboration  in  joint  asset  and  logistics  planning  activities 

W3C  SOAP 

for  multiple  systems  and  organizations.  The  intent  is  that 

BPMN 

each  native  local  system  will  provide  adapters  to  interact 

BPEL 

with  a  set  of  common  AAP  planning  services.  The  generic 

DEX 

interactions  to  the  AAP  web  services  are  specified  with 

EDI 

WSDL  soap  messages,  while  the  service  itself  is  described 
by  WSDL. 

The  data  exchanged  in  the  AAP  services  is  defined  appropriate  to 
the  type  of  service  and  the  content  specified  by  AAP  PLCS 
Activities  catalog,  AAP  Business  DEX  messages,  and  AAP 
EDI  content  to  support  the  BPEL  processes  and  the  BPMN 
process  message  synchronization. 

One  of  the  services  supports  access  to  the  UDDI  Logistics  and 

Asset  Directories 

UDDI  APIs 

#4:  Building  Blocks 


■  COTS  &  GOTS  Repository 

■  Building  Block  GUI  and  Algorithm 

■  Impartial  3rd  Party  Certification 


NCOIC  aides  customers 
in  achieving  design 
synthesis  &  design 
verification  via  the  work 
of  the  Building  Blocks 
(BB)  Functional  Team 

-  BB  database  is  a  public 
catalog  of  pattern-compliant 
building  blocks  available  for 
inquiry  by  member  and  public 
entities 

-  BB  self-verification  criteria 
for  candidate  re-usable  off- 
the-shelf  products 


Past  Present  Future 


V 

Where  is  it  going? 


Integration  of  products  of 
interest  to  NATO  will 

•SCOPE  -  characterize  interoperability  dimensions  increase  the  efficacy  of 

•NIF  (v2)  -  patterns  &  guidance  for  potential  solutions  the  BBdb. 

•BBdb  -  catalog  of  NiF-Compiiant  ots  products  Products  achieving 

•NCA  T  -  assessment  of  reaching  interoperability  goals  certification  wi  1 1  rei  nforce 

NCOIC  value  chain 


The  Building  Blocks  Perspe 


Building  Blocks  are  an 
Integral  part  of... 

The  overall  NCOIC 
tool  suite 


NCOIC 


Follow  &  Implement 
NCOIC  Patterns 


A  Building  Block  is  a  product  or  service 
that  implements  the  standards  &  guidance 
specified  in  NCOIC  Pattern (s)  to  enable 
specific  network-centric  capabilities 
for  a  set  of  intended  uses 


What  are  the  problems 
that  the  NCOIC  is  solving?. 


■  The  acquisition  community  wants  to  know  how  (and  to 
what  extent)  vendors’  offerings  may  work  together 

■  Vendors  need  to  understand  how  their  products  and 
services  may  be  used  in  network-centric  systems 
needed  by  the  overall  customer  community 

■  Both  should  recognize  which  standards  and  guidance 
to  use  in  order  to  assure: 

-  Desired  network-centric  capability 

-  Interoperability  between  and  among  other  products 


Building  Blocks  help  solve  these  problems 
with  real  products  and  services  that  can  be  effectively  used 
to  achieve  network-centric  capabilities 


What  Are  Building  Block 


■  A  Building  Block  is: 

-  A  product  or  service  that  implements  the  standards  and 
guidance  specified  in  NCOIC  Pattern(s)  to  enable  specific 
network-centric  capabilities  for  a  set  of  intended  uses 

■  Building  Blocks  ARE  NOT: 

-  An  architecture 

-  A  stand-alone,  complete  solution 

-  A  self-proclaimed  sales  pitch 

-  Future  “vaporware”,  promised  but  not  yet  available 


Value  of  Building  Blocks:  They  identify  real  products  or  services 
that  enable  specific  network-centric  capabilities  in  order  to  use 

them  with  confidence 


The  Value  of  Building  Bl 


Building  Blocks  help  to  match  Buyer  and  Supplier  Expectations 

■  Provides  a  registry  of  real  products  and  services  that  allows 
procurement  activities  and  system  integrators  to  identify  which 
items  meet  the  NCOIC  criteria 

-  A  means  for  products  to  be  visible  across  multiple  functional  areas 
and  markets 

■  Provides  a  Certification  and  Trademarking  program  to  promote 
the  identification  and  procurement  of  conformant  network-centric 
components  and  services 


Our  customers  are  asking  for  NCOIC  guidance- 
Building  Blocks  provides  this 


Building  Blocks  Promote  NCOIC- 
Compliant  Off-The-Shelf  Product! 


NCO  Initiatives  Database 
SCOPE  Model 


Typical  Process  Steps  to  Solutions: 

1.  Analysis  of  Alternatives 

2.  Requirements  Derivation 

3.  Requirements  Validation 


NCOIC  Interoperability  Framework  (NIF™) 


4.DESIGN  SYNTHESIS 

t 


The  NCOIC  Building  Blocks  process: 

-  Vendors  offer  &  describe  candidates 

-  Vendors  complete  Certification  Process 


B 

B 


-  Vendors  granted  rights  to  use 
“NCOIC  Certified”  logo 

-  Conformant  BB  listed  in  the  BB  Database 

-  Architects  use  certified  products  in  system 
designs 


5.  DESIGN  VERIFICATION 

6.  Deployment 

7.  Support 

8.  Upgrade  or  Disposal 


Supports 
Layered 
Quality 
of  Service 


Network 

Centric 

Assessment 

Tool 

(NCAT™) 

Building 

Blocks 

(BB) 

(details  on  next 
few  pages) 


Building  Blocks  Implement  NCOIC 
Patterns:  Standards  &  Guidance 
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Subject  Matter 
Expert 
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Operational 

Analysis 


Enterprise/System 

Architect 
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Architectural 

Analysis 


Technical 
Subject  Matter 
Expert 


Technical 

Analysis 


NCOIC 
Focus: 

Net-Centric 
Interoperability 
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NCO 
Requirements 


Overarching 

Guidance 


Operational 
(Domain) 
Patterns 

Integrated  Project 
Teams  (IPTs)  + 
SCOPE  Model 


Capability 

Patterns 


Technology 

Guidance 

0 


Technical 

Patterns 


Plus  NIF 

Overarching 

Guidance 


Plus  Specialized 
Frameworks 


Vendor  Products  &  Services  follow  &  implement  NCOIC  Patterns 


The  Benefits  of  Building  Bio 


■  Exposes  products  to  a  broader  market  base 

■  Promotes  entry  into  new  Network-Centric  markets  with  specific  products 
and  services 

(from  a  Product  Manager’s  perspective) 

■  Reduces  risk  in  all  phases  of  the  capability  acquisition  lifecycle 
(including  use  of  vendor  products  in  network-centric  system  designs) 

■  Potential  business  value  of  reducing  cost  and  risk  of  certification  effort 

■  Adds  focus  to  standards  compliance  strategy 

■  Accelerates  implementation  of  network-centric  solutions 

■  Provides  NCOIC  guidance  for  use  in  procurements 


Helps  all  stakeholders  to  achieve  the  benefits  of  the  NCOIC  Patterns 


Sample  Logistics  Building 
Block  Repository 


STANDARDS: 

■  SCOR  Supply  Chain  Operations  Model 

■  OMG  Business  Process  Modeling  Notation 

■  AAP  UDDI  (Log  Asset  Supplier  Directory) 

OASIS  WS  BPEL  (Business  Process  Execution  Language) 

■  OASIS  Business  DEX  (Data  Exchange) 

AAP  WSDL  (Web  Services  Description  Language) 

■  EDI  (Electronic  Data  Interchange) 

■  Others 

PRODUCTS  AND  SERVICES  (NOTIONAL): 

CDM  ICODES  (Integrated  Cooperative  Decision  Making) 

■  Transcore  eZGO  and  3sixty 

■  Hewlett  Packard  Real  Time  Enterprise  ZLE 

■  US  TRANSCOM  GTN  (Global  Transportation  Network) 

■  Others 


Way  Forward 


■  Unite  Diverse  Logistics  Communities  of  Interest  Stakeholders  by 
Leveraging  the  NCOIC  Processes  and  Tools. 

■  Further  develop  the  Logistics  Standards  Framework  in  union  with 
DOD,  NATO,  Commercial,  and  other  Stakeholders. 

■  Develop  remaining  identified  Patterns  for  the  global  logistics 
application  domain. 

■  Certified  products  for  the  Global  Logistics  Products  and  Services 
Repository. 


SUPPORT  SLIDES 


Building  Blocks  Certification 


■  "NCOIC  Certified"  logo  on  a  product  or  service 

-  Gives  buyers  assurance  that  vendor  promises  of  “network-centric  capabilities”  are  backed  up  by 
specific  conformance  to  NCOIC  Patterns 

-  Allows  conforming  vendors  to  advertise  this  assurance  to  their  customers  while  ensuring  that 
non-conforming  vendors  cannot 

-  Does  not  change  existing  company  and  industry  certification  programs 

Vendors  complete  an  application  process  to  certify  products  and  services  against 
the  specifications  in  an  NCOIC  Pattern 

-  NCOIC's  Certification  Authority  reviews  application  for  completeness 

-  If  OK,  then  the  product  or  service  is  listed  as  being  certified  in  the  Building  Blocks  database 

-  A  formal  challenge  process  allows  anyone  to  dispute  a  particular  vendor's  compliance  claim 

-  Vendors  must  enter  into  a  Trademark  License  Agreement  to  use  the  "NCOIC  Certified"  logo 

Architects  and  designers  consult  the  NCOIC  Building  Blocks  database  for  NCOIC 
Certified  products  and  services 


■  We  have  several  NCOIC  Operations  Patterns  in  work,  e.g.: 

-  For  Sense  &  Respond  Logistics:  Asset  Allocation  Planning  (AAP) 

-  For  NATO/Coalition:  Friendly  Force  Tracking  Interoperability  (FFTI) 

-  For  Emergency  Response:  Hastily-Formed  Networks 

We  anticipate  that  many  Technical  Patterns  will  be  developed  to  support  these 
and  other  operational  domains 

■  Implement  pilot  process  for  Building  Blocks 

-  Prior  demos  and  discussions  about  Building  Blocks  database, 
now  ready  for  actual  use 

-  Vendors  to  vet  the  above  patterns  and  associated  standards  by  submitting  candidate  products  into  the 
BB  process 

-  Acquisition  community  feedback  on  how  Building  Blocks  benefits  the  acquisition  process 
Incorporate  “lessons  learned”  to  improve  the  BB  process 


Value  Add...  if  you  so  choose 


GLOBAL  COMMERCIAL  AND  GOVERNMENT, 

COTS  AND  GOTS, 

HARDWARE  AND  SOFTWARE, 
PRODUCTS. 


Applying  the  Tenets  of  Military 
Planning  and  Execution  to 
Project  and  Systems 
Engineering  Management 


Systems,  Software,  and 
Solutions  Operation 


Tony  Lindeman,  PMP 
Senior  Systems  Engineer 

SAIC 

philip.a.lindeman@saic.com 


In  preparing  for  battle,  I  have  always 
found  that  plans  are  worthless,  but 
planning  is  indispensable.  ” 

General  Dwight  D.  Eisenhower 
34th  President  of  the  United  States 


Provide  aspiring  Systems  Engineers 
with  insight  into  how  basic  tenets  of 
military  planning  and  execution  can 
be  used  to  plan  and  monitor  the 
successful  execution  of  a  project. 


Terminology  to  Represent  Generic  Systems 

Engineering  Processes 


•  Technical  Management 

Processes 

-  Decision  Analysis 

-  Technical  Planning 

-  Technical  Assessment 

-  Requirements 
Management 

-  Risk  Management 

-  Configuration  Management 

-  Technical  Data 
Management 

-  Interface  Management 


•  Technical  Processes 

-  Requirements 
Development 

-  Logical  Analysis 

-  Design  Solution 

-  Implementation 

-  Integration 

-  Verification 

-  Validation 

-  Transition 


Communicate  the 
overall  objective  in 
general  terms  and 
leave  the  detailed 
planning  to  lower 
echelons 

Centralized 

planning; 

decentralized 

execution 


CAPTURE 

27lh  INFANTRY 


Rf=  1:50000 
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Commander’s 

Intent 

Tactical 

objective(s) 

Prioritization 

Success  criteria 


•  Logistics 

•  Contingency 
plans  based  on 
risk  assessment 

•  Communication 
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Big  picture,  puzzle  solvers 

Decomposition,  flowdown,  allocation,  and 
traceability 

Hierarchal  mindset 

-  Organization 

-  Specifications 

-  WBS 

-  Risk 

-  Communication 

Rigor  and  discipline  do  not  stifle  creativity 
Mathematically  inclined  -  “work  has  volume” 


•  Iterative  process 

-  Inputs  Decisions  ^  Outputs  + 
Assessment 

-  Ensuring  effort  is  value  added 

•  Current  Operations  and  Future 
Operations 

•  Reallocate  resources  as  battlefield  is 
shaped  and  evolves 

•  Maintain  momentum  of  keeping  overall 
effort  moving  forward 


From  Science  to  Solutions 


Reluctance  to  expend  significant  effort 

-  Playing  field  is  constantly  changing 

-  Obsolete  as  soon  as  it’s  put  into  place 

Types  of  planning 

-  Rough  Order  of  Magnitude  (ROM)  planning 

-  Initial  baseline  planning 

-  Re-baseline  planning 

-  Contingency  planning 

Baseline  plan  vs.  roadmap 

-  Detailed  plans  vs.  convergence  of  effort 

-  Precision  vs.  general  direction 

-  Know  when  to  focus  on  the  specifics  vs.  generalities 


Breaking  down  what  appears  to  be  an 
insurmountable  challenge  into  manageable 
and  achievable  activities 

*  Iterative  process  between  detailed  scheduling 
of  tasks  and  achieving  intermediate  objectives 

*  Identifying 

-  Program  milestones 

-  Key  Decision  Points  (KDPs) 

-  Technical  reviews  and  milestones 

*  Measuring  progress  in  terms  of  pre-defined 
success  criteria  and  demonstrating 
intermediate  capability 

Obtain  excruciating  scrutiny  and  eventual  buy-in 


From  Science  to  Solutions 
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Mission  planning  and  briefing 

-  Objective(s) 

-  Success  criteria 

-  Contingency  plans  for  risks  and  emergencies 

Pilot  mentality 

-  Power  required  can  exceed  power  available 

-  Running  out  of  fuel  can  ruin  your  day! 

-  Maintaining  altitude  and  airspeed  with  constant  power 
setting 

•  Scan  and  crosscheck  instruments 

•  Small  and  minimal  control  inputs  vs.  jerky  and  erratic 

Threat  Missiles  inbound  -  how  do  decisions  get 
made! 

-  Having  sufficient  data  and  information 

-  Timeliness 

Holding  forces  in  reserve 

-  Deploy  to  exploit  or  counter  a  threat 

-  Establish  criteria  for  deploying  when  and  how  much 
Expending  too  much  on  real-time  monitoring 


Science  to  Solutions 


“Systems”  thinking 

Planning  -  scrutiny  &  buy-in 

Command  Center  -  Segment  Current  & 
Future  Operations 

IMS  vs.  Roadmap  focus 

Making  timely  and  effective  decisions 


•  Monitor  and  measure  execution  in  order  to 
efficiently  and  effectively  apply  minor 
course  corrections 

•  Manage  reserves  to  exploit  opportunities 
and  repel  threats 


Cross-Command  Collaboration  Effort  (3CE) 


3  October  2008 
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Purpose  and  Agenda 


*  Purpose:  Provide  information  on  the  Cross  Command 
Collaboration  Effort  (3CE). 

*  Agenda: 

-  Background  and  Overview 

-  Capability 

-  Network 

-  Knowledge  Repository 

-  Requirements  Identification  and  Decomposition 

-  Systems  Engineering  Process 

-  Documents:  Processes  and  Procedures 

-  Planning:  Cross  Command  M&S  Investment  Strategy 

-Application 

-  FCS  Spin  Out 

-  Tools 

-  Summary  and  Way  Ahead 

Approved  for  Public  Release  -  Case  GOVT  08-8101 


What ...  3CE  Mission  and  Intent  -  Unique  Capability 


Mission:  Develop  a  cross  command  Army  M&S  and  data  environment  for  design, 
development,  integration,  and  testing  of  capabilities,  systems,  and  prototypes. 


JCIDS 


£ 


Concept 

Refinement 

Concept 

Decision 


Technology 

Development 


System  Development 
&  Dem^|tration 

Critical  Design  Review 


IOC 


FOC 


Production  &  Deployment 


LRIP/IOT&E 


FRP 

Decision 

Review 


Operations  & 
Support 


1  JRQC  Pre -Systems  Acquisition 


Systems  Acquisition 


Sustainment 


RDECOM 


tools,  data,  and  business  processes  that  provide  interoperable 
connectivity  that  links  the  participating  organizations, 
to  include  providing  a  common  3CE  environment  and  expertise 
for  the  Army  to  leverage. 

End  State:  A  3CE  environment  that  meets  the  common  requirements  of  all  three 
commands  and  Army  PMs  to  conduct  distributed  DOTMLPF  development. 
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Program  Life  Cycle  Support ...  a  Paradigm  Change 

The  3CE  is  focused  on  identifying  and  developing  an  M&S  environment  that  meets 
the  common  requirements  of  all  three  commands  and  PM  FCS  BCT  to  conduct 

distributed  DOTMLPF  development. 


Current  Operations 

There  are  numerous,  independent 
command  analytic  activities. 

MoM  decomposition  is  a  basic  component 
of  the  analytic  process 


Future  Operations 

Integrated  analytic  activities  conducted 
IAW  standard  operating  procedures. 


A 


A 


3CE  is  the  agent 
of  change 


PM  FCS  may  leverage  some  of  these 
activities  in  support  of  time  sensitive, 

program  specific  decisions. 


PM  FCS  leverages  3CE  to  develop  a 
reusable  M&S  environment  that  is  capable 
of  supporting  life  cycle  program  decisions. 


The  independent  activities  lead  to  discrete 
tool  (e.g.  M&S)  capability  development; 
sufficient  to  satisfy  the  immediate  decision 
requirement  and  often  not  capable  of 
supporting  reuse  requirements 


Standard  processes,  procedures,  and  a 
common  M&S  environment  provides  the 
means  to  conduct  more  integrated  and 
collaborative  DOTMLPF  development. 


Approved  for  Public  Release  -  Case  GOVT  08-8101 
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3CE  ...  How  is  This  Effort  Different? 


DoD/Arm y  Modeling  and  Simulation  Communities 


Acquisition  Analvak  Planning  Tossing  Training  Expariinentation 
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ATSL  PASE/JS 
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Army 
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•Organized  by  Communities 


ovemance  structured  to  support  the  Communities^ 
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s-Cutting  Tools 


Current  Domains 
•ACR 
•  RDA 
•TEMO 


Problem:  At  DoD,  Joint,  and  Service  level,  there 
exist  M&S  concepts,  strategies,  and  policies  to 
enable  an  “integrated  M&S  and  data  vision” ... 

no  resources  are  allocated  for  delivering  an 

integrated  solution  set  across  the  program 

lifecycle  -  currently  have  stovepipe  solutions  for 
S&T,  experimentation,  analysis,  testing,  and 
training. 


S&T 


CO  A:  To  fill  the  “gap  of  integrated  solution 
application,  ”  3CE  enables: 

•  Horizontal  integration  across  M&S  communities. 

•  “Fuzed  application”  of  M&S  and  data  solutions. 

•  Program  lifecycle  application 

•  Cooperative  implementation  -  3CE  Network. 


'  . ; 


Experimentation 


Analysis 


I  JGILJS 


Testing 


Training 


] 

Analysis 

1 

Testing 

1 

Training 
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Envisioned  Benefits  of  3CE  to... 


The  Army 

•  Provides  consistent  representation  through  common  tools  and  data  IAW  established 
standards  and  best  practices. 

•  Provides  the  capability  to  leverage  a  single  event  for  multiple  purposes. 

•  Provides  and  develops  environment  capabilities  that  are  traceable  to  user  needs  and 
design  requirements. 

•  Enhances  current  M&S  capabilities  and  reuse. 

•  Provides  a  leave  behind  capability  to  support  future  SoS  acquisition  programs. 


ATEC 


RDECOM 


Provides  a  consistent  •  Enables  consistent 
environment  for  M-T-M  data  from  field  tests 


TRADOC 


•  Enables  VV&A  to  test 
standards  for  M-T-M 


Program 


•  Provides  a  single  POC 
for  GFX  selection 


Reduces  preparation 
time  for  a  test 

Provides  reusable  and 
consistent  metrics 
from  development  to 
test 

Enhances  training 
proficiency  on  test 
equipment 


Reduces  the  number 
of  data  requests 


•  Reduces  time  to  obtain  •  Leverages  command 


characteristic  data 
from  the  program 


■  ■  ■  ■  ■  ■  ■  a  ■  hi  aw  w*  a  w  a  v*  a  a  a 

Enables  leveraging  r  9 

operational  capabilities  •  Leverages  multiple 
for  engineering  and  events  for  training 
performance  tests  _  .  .  . 


•  Provides  a  single 
environment  for 
analysis,  test,  and 
training 


events  for  multiple 
purposes 

Reduces  the  M&S  and 
data  coordination 
requirements 

Reduces  funding  for 
duplicative  M&S 
efforts 


Approved  for  Public  Release  -  Case  GOVT  08-8101 
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Roles  of  3CE 


3CE  will: 

•  Support  FCS  program  acquisition 
decisions. 

•  Enable  AETF  application. 

•  Assess  current  capabilities  to 
satisfy  requirements;  identify 
potential  M&S  solution  providers 
and  capability  gaps. 

•  Integrate  and  configuration  manage 
M&S  capabilities  that  are  common 
across  commands  into  the  Bliss- 
WSMR  LVC  environment. 

•  Provide  a  means  to  collaborate 
cross-command  and  cross-domain 
capabilities. 

•  Establish  and  share  a  set  of 
standards,  best  practices,  and 
expertise. 

•  Provide  a  leave-behind  capability  for 
future  analytic,  training,  and  testing 
support  to  acquisition  programs. 


3CE  will  not: 

•  Replace  a  command’s  unique 
mission  roles  and 
responsibilities. 

•  Replace  a  command’s  unique 
M&S  capabilities. 

•  Replace  a  command’s  unique  data 
capabilities. 

•  Impose  3CE  capabilities  on 
command  unique  missions. 

•  Operate,  maintain,  or  manage  a 
command’s  distributed  network. 


As  the  integrator  of  an  environment,  3CE  focuses  on  common  and  consistent 
capabilities  to  enable  cross  command  collaboration,  synergy,  and  reusability. 


Approved  for  Public  Release  -  Case  GOVT  08-8101 
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Purpose  and  Agenda 


*  Purpose:  Provide  information  on  the  Cross  Command 
Collaboration  Effort  (3CE). 

*  Agenda: 

-  Background  and  Overview 

-  Capability 

-  Network 

-  Knowledge  Repository 

-  Requirements  Identification  and  Decomposition 

-  Systems  Engineering  Process 

-  Documents:  Processes  and  Procedures 

-  Planning:  Cross  Command  M&S  Investment  Strategy 

-  Application 

-  FCS  Spin  Out 

-  Tools 

-  Summary  and  Way  Ahead 
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3CE  Network 


•  Network 

-  Established  3CE  network  consisting  52  total  nodes  built 
from  4  “Command”  networks 

-  TRADOC-  BLCSE  (Battle  Labs  and  Analysis  Centers) 

-  ATEC-  ATIN  (Test  Centers) 

-  RDECOM-  DVL  (Research  labs) 

-  PM  LSI  -  Sys  of  Systems  Integrated  Labs 

-  Provides  capability  to  conduct  distributed 
experimentation,  testing  and  analysis. 

-  Extensible  to  other  activities 

-  Provides  collaboration  services 

-  VTC 

-  Voice  over  IP 

-  Data  and  file  storage 

-  Persistent  Network  available  24/7/365 


Updating  Network  MOAs  and  Accreditation  to  support  select  Multi  National  participants  in  Army  directed 

events. 
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9 


3CE  Network  ...  A  Proven  Capability 


Boeing  -Seattle,  WA 


EPG 

Ft.  Lewis,  WA 


BA  E -Santa  Clara,  CA 


Huntington  Beach,  CA 


Boeing 
Mesa,  AZ 


i  YPG 
Yuma,  AZ 


The  3CE  network  is  a  proven  capability  that  has  demonstrated  stress: 

•  Collaborative- facilitated  SOI  planning  and  3CE  characterization  activitie 

•  Distributed  -  enabled  distributed  testing  for  8  SOI  integration  events 

•  Persistent-  maintained  greater  than  99%  availability 

•  Secure  -  accreditated  on  the  DREN  and  supported  numerous  events 

•  Extensible  -  linked  BLCSE,  ATIN,  DVL,  andSoSIL  environments 


BA  E -Minneapolis,  MN 

*  GDLS -Sterling  Heights,  Ml 

it  ARDEC 

cO 

Warren,  Ml 


DPG 

Dugway  UT 


TRAC,  BCBL  (L) 

Ft.  Leavenworth,  KS 


Honeywell 
Albuquerque,  NM 


BCBL  (H),  EPfcfe 
Ft .  Huachuca,  AZ  AMD  BL  O 

Ft.  Bliss,  TX 


#  LSI  /  Partner  Sites 
O  TRADOC  (BLSCE) 

•  ATEC(ATIN) 

O  RDECOM  (MatrexDVL) 
Network  Peering  Point 


Natick  Soldier  Center 
Natick,  MA 

o 

TAFtnEC ^  Picatinny  ArsenaL  NJ  C|T  CERDEC  /  M 

.  ^V^Ft.  Monmouth,  NJ 

9  Boeing 

cmr  Philadelphia,  PA 

Vienna  DTC.ATC,  ARL 

Boeing  uuqJlfSSV  APG-MD 

St.  Louis,  MO  ff'lelvo^VA^C^'^TEC 
•  UAMBLO  TRAC>r  Q  Alexandria,  VA 

MSBL  Ft.  Knox,  KY  Ft.  Lee^VA  \SDFC 

Ft.  Leonard  Wood,  MO  Ft.  Monroe,  VA 

ARSOBL 
Ft.  Bragg,  NC 

3DROSMDBL,  RTTC,  AMRDEC 
Huntsville,  AL  *0  BCBL  (G) 

'-^SBL(CSL,VSL) 

_  AMBL  "■ Bennln0'  GA 

CTSF/OTC®  Ft.  Rucker,  AL 

Ft.  Hood,  TX 
Overwatch  ^ 

Austin,  TX  *  a  STTC  O 

Orlando,  FL 

Boeing -Houston,  TX 


DSABL  _ 
Ft.  Sill,  OlO 

TRAC,  IRCCJDTCC 
White  Sands,  NM 


Ft.  Gordon,  GA 
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Why  the  Need  for  a  Knowledge  Repository  (KR)? 


The  3CE  KR  is  needed  by  multiple  users  to  enable  mission  execution  ... 


3CE  Team  Members  (Internal  Users) 


3CE  Mission  and  Intent 


-  Facilitate  team  member  coordination 

-  Enable  development,  test,  and  integration 
activities  relating  to  3CE’s  mission  and 
intent. 

-  Enable  Collaboration 

-  Enable  Document  Sharing 

-  Establish  processes  and  procedures  to  ensure 
KR  contains  current  and  validated  information 


3CE  Commands  &  PMs  (External  Users) 

-  Satisfy  information  needs 

-  3CE  capabilities  (tools,  network) 

-  3CE  processes 

-  3CE  data 

-  Satisfy  event  coordination  needs 

-  Support  the  planning,  development,  execution, 
and  reporting  of  events 

-  Facilitate  cross  command  data  visibility  and 
accessibility 


Mission  (Vision).'  Develop  a  cross  command  ArmyM&S  and  data 
environment  for  design,  development,  integration ,  and  testing  of 
capabilities,  systems,  and  prototypes. 


jcids  «■ 


At 

Corner 


A- 


jfi&p  fcijUrafc/i 


.  De^lopmefll 

iiir^ilr.kfi 

aimnaq&Jftyiftf 


JflL 


Prodiubw  &  Dettori 
UtlPflQTiE 


LIU 


'mm* 


Systems  AfflUflMfl 


?CS7.C?«Prqauc)s 


Intent: 


Purpose:  Identify,  develop,  and  maintain  a  core  set 
of  MSS  tools,  data,  and  business  processes  that  provide 
interoperable  connectivity  which  links  the  participatin 
organizations,  to  include  providing  a  common  BCE 
environment  and  expertise  for  the  Army  to  leverage, 


Relevant  Today  and  Info  the  Future 


End  State;  A  BCE  environment  that  meets  the 
common  requirements  of  all  three  commands  and  PM 
FCS  BCT  to  conduct  distributed  QCTMLPF  development 


...  its  demand  has  a  proven  reputation  for  enabling  mission  success. 
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How  do  We  Function?  ...  3CE  Overarching  Process 


The  overarching  process  describes  how  3CE  will  execute  its  mission ... 


Source  of  Requirements 


x 


FCS  SSP,  FACTS, 
M&S  Community, 
PM/MAT  DEV,  DTE, 
SIMEX,  LUT,  L-V-C, 
etc... 


Research  Requirements 
Command  Requirements 
PM  Requirements 


User  (3CE  KR) 
Facilitate  Event  Planning 
Facilitate  Event  Execution 
Facilitate  Accreditation  & 
Certification 
Leverage  3CE  Toolkit 
through  the  3CE  KR 

*  Event  Requirements 


Requirements: 

*  High  Level 

*  Analytical 
Basis 

*  Across 
Commands 


Legend 

Inputs 

Outputs 

Process 


List  of  Configuration  Managed 
M&S  Solutions 
Technical  Capabilities 
Business  Processes 
Standards 

l _ 

Approved  for  Public  Release 


Capability  Development  and  Integration 


3CE  Program  Management 


■  3CE  TFA  IPT - 


Analysis/E  valua  tion 

Identify  Requirements 

Consolidate 

Archive 

Verify 

Prioritize 

Decompose 

Refine 

I 


Prioritized 

Requirements: 

•  Analytical 

•  Other 

- ► 

List  of  M&S 
Capability  Gaps 


System  Engineerinci 
Assess  and  define  M&S 
Requirements 
Identify  M&S  Capability 
maturity  levels  and 
“gaps” 

Refine  Capabilities 
Development  Road  Map 
Update  Knowledge 
Repository 


Prioritized  List  of  M&S 
Capability  Gaps 


Infrastructure,  Integration,  & 

Verification 

•  Validate  and  Verify 

Solutions 

Solutions 

*  Integrate  M&S  Solutions 

*  Configuration  Manage 

Solutions 

*  Manage  Current  Capabilities 

Technical  Development 

Identify  Current 

Capabilities 

Design  M&S  Solutions 

Develop  M&S  Solutions 

Develop  Technical 

Solutions 
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Analyst/Evaluator  Requirement  Decomposition 


Authoritative  Source  Documents 
for  A/E  Requirements 


Decomposition  Process 


r 


ATEC  Requirements 
RDECOM  Requirements 
TRADOC  Requirements 
PM  Requirements 


Database  of  Prioritized 
A/E  Requirements 
(DOORS) 


•  Study  Issues  /  EEAs 

•  Dimensional 

•  Source 

|\  •  Research  Requirements 

k 

Parameters 

k 

•  Location 

\  •  Development  Requirements 

y  mops 

\ 

•  Time 

■  /  •  Evaluation  Requirements 

— 1/ 

/  •  MOEs 

H 

•  Frequency 

’  •  Test  Requirements 

-  Analwtin 

1 

•  MOFEs 

1 

•Fidelity 
^ - 

V. 


Requirements  Measurement  Space 


Elements 


Provide  measures 
to  Users 


Legend 

Inputs 

Outputs 

Process 


Intended  Usage 
Measurement  Calculations 
Scenario  Requirements 
AV-2  Dictionary 


DCMP 


Enable  M&S 
capability  gap 
determination. 


Refined 
requirements  from 
user  planning  and 
execution. 


Refined 

data 

element 

fidelity. 
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Requirement  Types 


User  A/E  M&S 


Requirements 


CDD/ORD 

Requirements 


Ends 

Verify  that  the 
material  solution 
must  be  capable  of 
detecting  a  minefield 
90%  of  the  time. 

Mission  Need 


Requirements 


Analytic 

Requirements 


Ways 


•  MOE/MOP 

>#  of  mines  detected 
>%  of  mines  detected 
>#  systems  destroyed 
>%  of  systems  destroyed 

Capability 


Requirements 


Capability 

Requirements 


Means 


The  test  system 
shall  simulate 
minefield  detection 
and  breaching. 

Requirement 


What  “requirements”  are  we  identifying? 


Approved  for  Public  Release  -  Case  GOVT  08-8101 


14 


Tool  Selection  Using  the  3CE  Knowledge  Repository 


Program/event 

Requirements 

Input 


User  Queries 
(based  on  user  requirements) 


Technical  Requirements 
MOP  Context  Views 
OVs  and  SVs 
Supporting  Data 


"AS-IS"  tool  capabilities 


•  Capabilities 
aligned  to  user 
requirements 

•  Example 
requirements 

•  Products 
aligned  to  user 
requirements 
(analytical, 
operational,  & 
technical 

•  M&S  tool/data 
collaboration/ 
sharing 
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How  Do  We  Involve  the  M&S  Community? 


tyEndstate 


Q  Current  State 


M&S  Needs 

•  ATEC 

•  RDECOM 

•  TRADOC 
•PM/LSI 


FY07  PEP 


Tasks 
Products 
Metrics 


Decomoosed  AC  Needs 


•  Executable  Task  Level 

•  Product  Focused 

•  Interdependent 


FY07  Assessment 


*  Metric-based 

•  Task  Progress 

*  Task  Completion 

•  Product  Statu~ 


1 


AC  Need  Tasks  Assessment 


*  Aligns  Need  Tasks 
to  FY07  Tasks 

*  Assesses  Need  Tasks 

*  Refines  Status 


AC  Need  Assessment 


•  Applies  Task  Status 

•  Updates  Need  Status 

•  Facilitates  PRM 


Q  Future  State 


Cross  Command 
Collaboration  Effort  (3CE) 


FY08  Tasks 


•  PRM-Based 

•  Endstate  Focused 

•  Mutually  Supporting 

•  Commands 

•  PM 

•  OOS 

•  Network 

•  Infrastructu 


I 


\ 
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•  Needs-Based 

*  Status  Focused 

•  FY08-FY12 

*  Command/PM  Aligned 
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Purpose  and  Agenda 


*  Purpose:  Provide  information  on  the  Cross  Command 
Collaboration  Effort  (3CE). 

*  Agenda: 

-  Background  and  Overview 

-  Capability  and  Product  Accomplishments 

-  Network 

-  Knowledge  Repository 

-  Requirements  Identification  and  Decomposition 

-  Systems  Engineering  Process 

-  Documents:  Processes  and  Procedures 

-  Planning:  Cross  Command  M&S  Investment  Strategy 
-Application 

-  FCS  Spin  Out 

-  Tools 

-  Summary  and  Way  Ahead 
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Accomplishments  to  Date  for  FCS 


Event  Support 

-  Experiment  1.1 

-  Linked  over  3CE  sites 

-  Provided  live  video  and  AAR  for  experiment 

-  Ability  to  share  lessons  learned  throughout  Army  real  time. 

-  SO  simulation  federation 

-  Identified  requirements 

-  Identified  solutions 

-  Conducting  integration  to  provide  common  solution  to  4 
events. 

A  3CE  environment  that  meets  the  common  requirements  of  all  three  commands  and  PM 

FCS  BCT  to  conduct  distributed  DOTMLPF  development. 
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An  Applied  Example  ...  3CE  Supporting  SO 


3D  Viz 
Only 


A  Liv  -Virtual-  Constructive  Integrated 
Environment  Supporting: 

Training  (TRADOC) 

Technical  Field  Test  (LSI) 

Force  Design  Testing  and  Evaluation  (TRADOC) 
Limited  User  Test  (ATEC) 


AAR 

DCARS 
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Enabling  SOI  Integration  ...  Requirements  Focused 


Technical 
Requirements 
By  Activity 
(LUT) 


Underpins 
validity  of 
technical 
requirements 


f 


Analytic  Source 
Documents 
( i.e.  SEP,  DTP, 
DCMP) 


3CE  SV  -  4  (3CE  Framework) 


Informs  development  of 
exit  criteria  for 
integration  tests 
Evolves  integrated  M&S 
architecture  over  time 


•  Tracks  test  results 

•  Identifies  tested 
requirements 

Informs  Gaps 

Sets  expectations 

Informs  VV&A 


Technical 
Requirements 
by  Functionality 
(Threat,  DCA 
Communications, 
Scenario,  Event 
Management.) 


Links  activity-submitted  technical 
requirements  to  the  original  capability  needs 
and  discriminators  used  to  identify  the  core 
federation  architecture 


O 

> 

"0 

c/> 


Integration 

Test 

5  Schedule  3 


Potential  GAPs  of  technical  requirements 
will  inform  the  ability  to  achieve  analytic  requirements 
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Purpose  and  Agenda 


*  Purpose:  Provide  information  on  the  Cross  Command 
Collaboration  Effort  (3CE). 

*  Agenda: 

-  Background  and  Overview 

-  Capability  and  Product  Accomplishments 

-  Network 

-  Knowledge  Repository 

-  Requirements  Identification  and  Decomposition 

-  Systems  Engineering 

-  Documents:  Processes  and  Procedures 

-  Planning:  Cross  Command  M&S  Investment  Strategy 

-  Application  Accomplishments 

-  FCS  Spin  Out 

-  Tools 

-  Summary  and  Way  Ahead 
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Summary 


As  the  integrator  of  an  environment ,  3CE  focuses  on  common 
and  consistent  capabilities  to  enable  cross  command 
collaboration ,  synergy ,  and  reusability ... 

•  Provides  consistent  representation  through  common  tools  and  data  IAW 
established  standards  and  best  practices. 

•  Provides  the  capability  to  leverage  a  single  event  for  multiple  purposes. 

•  Provides  and  develops  environment  capabilities  that  are  traceable  to  user 
needs  and  design  requirements. 

•  Enhances  current  M&S  capabilities  and  reuse. 

•  Provides  a  leave  behind  capability  to  support  future  SoS  acquisition  programs. 

...  through  the  activities  in  support  of  SO  integration ,  3CE  will 
have  an  instantiation  of  this  capability  to  support  future  user 

activities  across  the  Army . 

•  Provide  a  core  federation  with  supporting  functional,  interoperability,  event 
management,  and  data  collection  and  analysis  tools. 

•  Provide  an  accessible  knowledge  repository  that  provides  the  processes, 
procedures,  standards,  and  expertise  to  leverage  3CE  capabilities. 

•  Provide  a  persistent  and  secure  network  that  enables  collaboration  and 
interoperability  across  the  commands  and  the  PM/LSI. 
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SE  Effectiveness  - 

Overview 


STRENGTH  THROUGH  INDLSTKV  4  TECHNOLOGY 


The  SE  Effectiveness  Survey 

Quantifies  the  relationship  between  the 
application  of  Systems  Engineering  best  practices 
and  the  performance  of  system  development  projects 

r-yv-i 

TODAY'S  OUTLINE 

Projects  with  better 

/  1.  Rationale  and 

Systems  Engineering 

Background 

capabilities  deliver  < 

2.  The  Challenge 

better  Project  \ 

s  3.  The  Rigor 

/  Performance!  \ 

4.  The  Results! 

5.  Conclusions  &  Caveats 
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Previous  Studies  -  Summary 


STRENGTH  T1IHOIGII INDISTRV  &  TECHNOLOGY 


STUDY 

APPLICABILITY 

Author  & 
Background 

Findings 

SE  Activities 

Definition  of 
Success 

Characteristics 
of  Project 

Gruhl  ( 1 992) 

32  NASA  Pgms 

8-15%  Upfront 
Best 

First  two  of  five 
development  phases 

Cost  (Less  cost 
overrun) 

Large;  Complex;  all 
NASA 

Herbsleb  (1994) 

13  CMM 
Companies 

Process 
Improvement 
ROI4.0-  8.8 

CMM  Process 

Areas 

Cost  (Cost 
reduction  through 

SE  investment) 

Various;  federal 
contracting 

Honour  (2004) 
Survey  INCOSE 
SEs 

15-20%  of 
project  should 
be  SE 

Overall  SE  level  of 
effort  (Cost)  & 
related  SE  quality 

Cost  &  Schedule 

Various  sizes 
(measured  by  total 
project  cost) 

Boehm  &  Valerdi 
(2006) 

COCOMO  II 

SE  importance 
grows  with 
project  size 

COCOMO  II  RESL 
(Architecture  and 
Risk) 

Cost 

Various  sizes,  but 
software  systems 
only 

Boehm  &  Valerdi 
(2004) 

COSYSMO 

Estimate 
within  30% 
effort  50%  - 
70%  of  time 

33  activities  defined 
by  EIA  632 

Cost 

Mostly  successful 
projects  from 
federal  contractors 

Ancona  & 

Caldwell  (1990) 

Boundary 

Management 

Managing  team 
boundary  15%; 
more  is  better 

Team  boundary 
activities  —  interface 
between  team  and 
external 

Product 

Performance 
(Successfully 
marketed  products) 

Technology 

products 

Frantz  (1995) 
Boeing  side-by- 
side  projects 

More  SE 
yielded  better 
quality  & 
shorter 
duration 

Defined  by  Frantz 

Product 

Performance  & 
Schedule  (Quality 
of  product  and 
duration  of  project) 

Three  similar 
systems  for 
manipulating 
airframes  during 
assembly 
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Does  this  sound  familiar? 


STRENGTH  TIIKOIGH  INDUSTRY  &  TECHNOLOGY 


The  SE  efforts  on  my  project  are 

critical  because  they  ... 

...  pay  off  in  the  end. 

...  ensure  that  stakeholder 

requirements  are  identified  and 
addressed. 

...  provide  a  way  to  manage 
program  risks. 

...  establish  the  foundation  for  all 
other  aspects  of  the  design. 

...  optimize  the  design  through 
evaluation  of  alternate  solutions. 


We  need  to  minimize  the  SE  efforts 

on  this  project  because  ... 

...  including  SE  costs  in  our  bid  will 
make  it  non-competitive. 

...we  don't  have  time  for  , paralysis 
by  analysis ".  We  need  to  get  the 
design  started. 

...we  don"thave  the  budget  or  the 
people  to  support  these  efforts. 

...  SE  doesn"tproduce  deliverable 
outputs. 

...  our  customer  won"t  pay  for  them. 


•These  are  the  ASSERTIONS,  but  what  are  the  FACTS? 
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The  Problem 


STRENGTH  T1IKOI  G1I  INDUSTRY  &  TECHNOLOGY 


It  is  difficult  to  justify  the  costs  of  SE  in  terms  that  program 
managers  and  corporate  managers  can  relate  to. 

•  The  costs  of  SE  are  evident 

-  Cost  of  resources 

-  Schedule  time 

•  The  benefits  are  less  obvious  and  less  tangible 

-  Cost  avoidance  (e.g.,  reduction  of  rework  from  interface 
mismatches) 

-  Risk  avoidance  (e.g.,  early  risk  identification  and  mitigation) 

-  Improved  efficiency  (e.g.,  clearer  organizational  boundaries  and 
interfaces) 

-  Better  products  (e.g.,  better  understanding  and  satisfaction  of 
stakeholder  needs) 
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The  Questions 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


•  How  can  we  quantify  the  effectiveness 
and  value  of  SE? 

•  How  does  SE  benefit  program 
performance? 
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The  Solution 


STRENGTH  T1IKOIG1I INDUSTRY  &  TECHNOLOGY 


Obtain  quantitative  evidence 
of  the  costs  and  benefits  of 
Systems  Engineering 
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The  Challenge  - 

SE  Effectiveness  Survey 


STRENGTH  T1IK01G1I  INDUSTRY  &  TECHNOLOGY 


Hypothesis:  The  effective  performance  of  SE  best  practices  on  a 
development  program  yields  quantifiable  improvements  in  the  program 
execution  (e.g.,  improved  cost  performance,  schedule  performance, 
technical  performance). 

Objectives: 

•  Characterize  effective  SE  practices 

•  Correlate  SE  practices  with  measures 
of  program  performance 

Approach: 

•  Distribute  survey  to  NDIA  companies 

•  SEI  analysis  and  correlation  of  responses 

Survey  Areas: 


■jj  Pong  I  I  I  I  FIT  Internet _ ^ 


Process  definition 


Trade  studies 


Project  reviews 


Project  planning 
Risk  management 
Requirements  development 
Requirements  management 


Interfaces 
Product  structure 
Product  integration 
Test  and  verification 


Validation 

Configuration  management 
Metrics 
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The  Rigor  - 

Followed  Planned  Lifecycle 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 
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The  Rigor  - 

Formally  Selected  Set  of  SE  Activities 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


CMMI-SW/SE  vi.i 

•  25  Process  Areas 

•  179  Goals 

•  614  Practices 

•  476  Work  Products 


Systems 
Engineering- 
related  Filter 


Size  Constraint 
Filter 


Considered  significant 
to  Systems  Engineering 


•  14  Process  Areas 

•  31  Goals 

•  87  Practices 

•  199  Work  Products 


•  1 3  Process  Areas 

•  23  Goals 

•  45  Practices 

•71  Work  Products 


Survey  was  developed  based  on  standards 
and  recognized  SE  experts 


Candidate  Methods: 

Case  Studies 


STRENGTH  T1IKOI  G1I  INDUSTRY  &  TECHNOLOGY 


Method  •  Establish  collaboration  with  one  (or  a  few)  defense 
contractor(s) 

•  Choose  a  few  completed  projects 

•  Collect  and  analyze  data  to  quantify  the  costs  and  benefits 
of  the  SE  applied  to  the  projects 

Pros  •  In-depth,  multi-faceted  study 


Cons  •  Reluctance  of  contractors  to  expose  sensitive  data 

•  Lack  of  data 

-  Consistency:  No  generally  accepted  definition  of  SE 

-  Availability:  1)  SE  efforts  not  often  budgeted  and  tracked 

2)  Benefits  of  SE  are  difficult  to  quantify 

•  Lack  of  generalization 

-  “That  doesn’t  apply  to  us;  we  do  it  differently.” 

-  “That’s  just  one  (or  a  few)  project(s).” 
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Candidate  Methods: 

Organizational  Survey 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


Method  •  Survey  defense  contractor  organizations 

•  Collect  and  analyze  data  to  quantify  the  costs  and  benefits 
of  SE  applied  within  the  organization 


Pros  •  Based  on  a  representative  sample  of  the  industry 


Cons  •  Reluctance  of  contractors  to  expose  sensitive  data 

•  Lack  of  data 

-  Consistency: 

1)  No  generally  accepted  definition  of  SE  across  organizations 

2)  Uneven  application  of  SE  within  organizations 

-  Availability: 

1 )  SE  efforts  not  often  budgeted  and  tracked 

2)  Benefits  of  SE  are  difficult  to  quantify 


SE  Effectiveness  Committee  -  Status 
October  21,  2008 


14 


Candidate  Methods: 

Project  Survey 

Method  •  Survey  individual  defense  contractor  projects 

•  Collect  data  on  the  application  of  selected  SE  practices 

•  Collect  data  on  the  overall  performance  of  the  project 

•  Analyze  results  to  identify  relationships  between  SE 
application  and  project  performance 

Pros  •  Based  on  a  representative  sample  of  the  industry 

•  The  survey  provides  a  common  definition  of  SE 

•  Project  performance  data  is  widely  available 

Cons  •  Reluctance  of  contractors  to  expose  sensitive  data 
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Implementation  of  the  Systems 

Engineering  Effectiveness  Survey  (SEES)  111/111 

Jr  \  /  STRENGTH  THROUGH INDUSTRY  &  TECHNOLOGY 


1.  Define  the  goal 

2.  Choose  the  population 

3.  Define  the  means  to  assess 
usage  of  SE  practices 

4.  Define  the  measured 
benefits  to  be  studied 

5.  Define  the  „other'Tactors  to 
be  studied 

6.  Develop  the  survey 
instrument 

7.  Execute  the  survey 

8.  Analyze  the  results 

9.  Report 

10. PIan  future  studies 


1  The  Effectiveness  of  Systems  Engineering:  A  Survey  -  Microsoft  Internet  ExpJpr 

JdxJ 

File  Edit  View  Favorites  Tools  Help 

t] 

Q**  -O'- 0  S 

Search  Favorites  4P 

-  f  S  *  * 

Address  |  rj  https :  //seir ,  sei ,  emu ,  edu/feedback/5ystemsEngineering Demo ,  htm  T 

Coogle 

\Qt  ;*|go  I 

T  Settings^ 

Software  Engineering  Institute 


UMl 


vriinirjii  in  wuhnnrm:  nc 


The  Effectiveness  of  Systems 
Engineering:  A  Survey 

14.  Approximately  what  percentage  of  non-recurring  engineering  (ME)  does  systems  engin 
approximate  percentage  -  without  the  percentage  sign) 


I  % 

Is  the  NEE  percentage  estimated,  or  is  it  a  measured  value?  f Please  select  one) 
f  Estimated 
^  Measured 


-i 


A 

l 

Done 

l^j  jj  Internet 

j 
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Population  and  Sampling 
Method 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


Population 

•  DoD  prime  contractors  and  subcontractors  who 
produce  products  (as  opposed  to  services). 

Sampling  Method 

•  NDIA  SE  Division  represents  a  reasonable  cross 
section  of  the  chosen  population 

•  Invite  all  product-supplying  organizations  within  the 
NDIA  SE  Division  to  participate. 

•  Random  sampling  within  each  organization 
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Assessment  of  SE  Practices  i 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


•Question  #1 

•What  SE  activities  do  you  apply  to  your  project? 
Challenge 

•  No  generally  accepted  definition  of  what  IS  and  what  IS  NOT  a  part 
of  SE. 

-  “How  much  SE  do  you  do  on  your  project?”  <£=  No  answer 

•  SE  is  often  embedded  in  other  tasks  and  not  budgeted  separately 

-  “How  much  does  your  project  spend  on  SE?”  <=  No  answer 

Solution 

•  Avoid  a  defining  SE 

-  Too  much  controversy 

•  Ask  about  the  results  of  activities  that  are  generally  agreed  to  be  SE 
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Assessment  of  SE  Practices  2 


STRENGTH  T1IK01G1I  INDUSTRY  &  TECHNOLOGY 


Based  on  CMMI-SE/SW  vl.1 

Focused  on  identifying  tangible  artifacts  of  SE  activities 

•  Work  products 

Work  Products  chosen  by  a  panel  of  SE  experts  from 
government,  industry,  and  academia 

•  First  pass  -  selected  CMMI  Work  Products  that  were  (in  the 
judgment  of  the  SE  expert  panel)  related  to  SE 

•  Second  pass  -  selected  SE-related  Work  Products  that  were 
(in  the  judgment  of  the  SE  expert  panel)  most  significant 
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Assessment  of  SE  Practices  3 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


•  CMMI- 
SE/SW/IPPD  vi.i 


•  25  Process  Areas 

•  179  Goals 

•  614  Practices 

•  476  Work  Products 


•Systems 
Engineering- 
related  Filter 


•Size 

Constraint 


•Considered  significant, 
to  Systems 
Engineering 


•  14  Process  Areas 

•  31  Goals 

•  87  Practices 
•199  Work  Products 


•  1 3  Process  Areas 

•  23  Goals 
•45  Practices 

•71  Work  Products 


•Survey  content  is  based  on  a  recognized  standard  (CMMI) 
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Assessment  of  SE  Practices  4 


STRENGTH  THROUGH  INDUSTRY  Ik  TECHNOLOGY 


Goal 

PRACTICE 

WORK  PRODUCT 

!* 

s 

Integrated  Project  Management  for  IPPD 

SGI:  Use  the  Projects 
Defined  Process  -  The 
project  is  conducted  using  a 
defined  process  that  is 
tailored  from  the 

SP  1 .1-1 :  Establish  the  Projects  Defined  Process  - 
Establish  and  maintain  the  projects  defined  process. 

The  projects  defined  process 

SP  1 .2-1 :  Use  Organizational  Process  Assets  for  Planning 
Project  Activities  -  Use  the  organizational  process  assets 
and  measurement  repository  for  estimating  and  planning 

Project  estimates 

Project  plans 

Goal 

PRACTICE 

WORK  PRODUCT 

SE  Work 

Product 

KEY  SE 

WP 

=a 

Q 

|sp  1.5-1:  Contribute  to  the  Oraanizational  Process  Assets  Proposed  improvements  to  the  organizational  process  assets 

/?' 

^ : 


Organize  Integrated 
for  IPPD  -  The 
Integrated  teams  needed  to 
execute  the  project  are 
identified,  defined, 
structured,  and  tasked. 


SP  4.1-1:  Determine  Integrated  Team  Structure  for  the 
Project  -  Determine  the  integrated  team  structure  that  will 
meet  the  project  objectives  and  constraints. 


Published  principles,  shai^S-Ts  _ 

and  objectives  (e.g.,  posters,  wallet  cards  publts; 
suitable  for  wall  hanging; 


Assessments  of  the  product  ar 
including  risk  and  complexity 


l:  Establish  Integrated  Teams  -  Establish  and 


I  product  architectures, 


m  structures  based  on  the  WBS  and  adaptations 


Alternative  concepts  for  integrated  team  structures  that  include 
responsibilities,  scope,  and  interfaces 


IS  for  each  integrated 


Proj05 

Proj06 


•Identified  as  key  SE  artifacts 

•Identified  as  SE 
artifacts 


Assessments  of  the  product  and  product  architectures, 
including  risk  and  complexity 

IyI 

Integrated  team  structures  based  on  the  WBS  and  adaptations 

Y 

Y 

ProjOS 

ProjOG 

Alternative  concepts  for  integrated  team  structures  that  include 
responsibilities,  scope,  and  interfaces 

Y 

Selected  integrated  team  structure 

Y 

S G  4:  0  rg  a  n  iz  e  Integ rated 
Teams  for  IPPD  -  The 
integrated  teams  needed  to 
execute  the  project  are 
identified,  defined, 
structured,  and  tasked. 


SP  4.1-1:  Determine  Integrated  “earn  Structure  for  the 
Project  -  Determine  the  integrated  team  structure  that  wil 
best  meet  the  project  objectives  and  constraints. 


|  relative  priority. 


J  L 
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Assessment  of  SE  Practices  5 


STRENGTH  T1IK01G1I  INDUSTRY  &  TECHNOLOGY 


•SE  Work  Products  chosen  in  the  following  CMMI  Process  Areas: 


CMMI  Process  Area 

#WP 

•  Organizational  Process  Definition 

OPD 

1 

•  Project  planning 

PP 

10 

•  Risk  management 

RSKM 

6 

•  Requirements  development 

RD 

8 

•  Integrated  Project  Management 

IPM 

3 

•  Requirements  management 

RM 

10 

•  Configuration  management 

CM 

7 

-c 

Trade  studies 

•  Technical  Solution 

TS 

13 

Interfaces 

•  Product  Integration 

PI 

1 

Product  architecture 

•  Verification 

VER 

10 

•  Validation 

VAL 

2 
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Assessment  of  Project 
Performance 


♦Question  #2 

•How  is  your  project  going? 


Address  TOTAL  Project 
Performance 

•  Project  Cost 

•  Project  Schedule 

•  Project  Scope 
Focus  on  commonly  used 
measurements 

•  Earned  Value  Management 
(CPI,  SPI,  baseline  management) 

•  Requirements  satisfaction 

•  Budget  re-baselining  and  growth 

•  Milestone  and  delivery  satisfaction 
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Assessment  of  Other  Factors 


STRENGTH  T1IROLGH  INDUSTRY  &  TEUTUNOLXKiY 


•Question  #3 

•What  other  factors  affect  project  performance? 


SE  Capability  is  not  the  ONLY  thing  that  can  impact  Project 
Performance.  What  about: 

•  Project  Challenge  -  some  projects  are  more  complex  than  others 

-  Lifecycle  scope,  technology  maturity,  interoperability  needs,  precedence,  size,  duration, 
organizational  complexity,  quality  of  definition 

•  Acquirer  Capability  -  some  acquirers  are  more  capable  than 
others 

-  Requirements  quality,  acquirer  engagement,  consistency  of  direction 

•  Project  Environment  -  projects  executed  in  and  deployed  to 
different  environments  have  different  needs 

-  Acquiring  organization,  user  organization,  deployment  environment,  contract  type,  developer’s 
experience,  developer’s  process  quality 
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Developing  the  Survey  Instrument: 

Requirements 


Target  Respondent 

•  Program  /  Project  Manager  or  designee  for  individual  projects 
Deployment 

•  Web  based 

•  Anonymous 

-  No  questions  eliciting  identification  of  respondent,  project, 
or  organization 
Target  Response  Time 

•  Average:  30  minutes 

•  Maximum:  60  minutes 
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Developing  the  Survey  Instrument: 

Questionnaire  Structure 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


Section  1  -  Project  Characterization 

•  Project  Challenge 

•  Acquirer  Capability 

•  Project  Environment 

Section  2  -  SE  Capability  Assessment 

•  Process  Definition,  Project  Planning  &  Risk  Management 

•  Requirements  Development,  Requirements  Management  & 
Trade  Studies 

•  Interfaces,  Product  Structure  &  Integration 

•  Verification,  Validation,  &  Configuration  Management 
Section  3  -  Project  Performance  Assessment 

•  Earned  Value  Management 

•  Other  Performance  Indicators 
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Developing  the  Survey  Instrument: 

Question  Formats  i 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


Quantitative  Questions 

•  Some  questions  require  numeric  answers 

-  What  is  the  current  total  contract  value  of  this  project? 

•  Other  questions  require  an  approximate  numeric  response 

-  The  schedule  of  this  project’s  critical  path  when  compared  to  the  current  IMS 
approved  by  the  acquirer  is: 

□  Greater  than  6  months  late 

□  Greater  than  3  months  late 

□  Greater  than  6  months  early 

Free  Form  Questions 

•  Provides  an  opportunity  for  the  respondent  to  enter  his 
thoughts 

-  What  performance  indicators  (beyond  cost  and  schedule)  have  been  particularly 
useful  in  managing  your  project? 
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Developing  the  Survey  Instrument: 

Question  Formats  2 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


Likert  Items 

•  Many  of  the  questions  assessing  SE  Capabilities  use  a  “Likert” 
format 

-  a  psychometric  scale  commonly  often  used  in  survey  research 

-  respondents  specify  their  level  of  agreement  to  a  statement 

“My  project  has  a  <work  products  with  <defined 
characteristics >” 

□Strongly  Disagree  DDisagree  DAgree  DStrongly  Agree 


•Example 

•This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan 
(IMP)  that  is  an  event-driven  plan  (i.e.,  each  accomplishment  is  tied  to  a 
key  project  event. 

•□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 
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Developing  the  Survey  Instrument: 

Testing  i 


Deployed  to  volunteers  among  the  organizations 
participating  in  the  development  of  the  survey 

Interviews  with  respondents  addressing: 

•  Understanding  of  the  questions 

-  Nearly  all  questions  interpreted  without  ambiguity 

-  Some  rewording  to  ensure  consistent  understanding 

•  Time  required  for  completion 

-  Typical  45  minutes.  Maximum  >2  hours 

-  Issues  with  questions  requiring  quantitative  inputs 

•  Suggestions  for  improvements 
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Developing  the  Survey  Instrument: 

Testing  2 


Questionnaire  revised  to  address  results  of  initial  testing 

•  Elimination  of  questions 

•  Replacement  of  pure  quantitative  questions  with  approximate 
quantitative  questions 

-  Selection  of  ranges  of  values  rather  than  the  entry  of 
numeric  values 

-  Provided  cues  for  the  level  of  detail  desired 

Redeployed  for  testing 

•  All  questions  interpreted  without  ambiguity 

•  Time  required  for  completion 

-  Typical  30  minutes.  Maximum  60  minutes 
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Survey  Deployment 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Challenges 

Solutions 

Ease  of  Participation 

•Method  of  response  must  be  easy  to 
encourage  maximum  participation 

•  Deployment  and  response  via  the 
internet 

Confidentiality 

•Many  NDIA  members  represent 
commercial  defense  contractors. 
•Proprietary  data  cannot  be  exposed 

•  Data  collection  and  analysis  done  by 
the  SEI.  Only  aggregated  results 
provided 

Anonymity 

•Further  protection  of  proprietary  data 

•  No  questions  soliciting  respondent, 
project,  or  organization  identification 

•  “blind”  authentication  for  survey  login 

Incentivization 

•  Respondents  and  their  organizations 
need  a  reason  (beyond  altruism)  to 
participate 

•  Respondent  solicitation  through 
company  management  hierarchy 

•  Early  access  to  survey  results  to 
support  benchmarking  and  process 
improvement 

SE  Effectiveness  Committee  -  Status 
October  21,  2008 


31 


Survey  Deployment: 

Respondent  Solicitation  i 


STRENGTH  T1IKOI  G1I  INDUSTRY  &  TECHNOLOGY 


Review  the  roster  of  “Active  Members”  of  the  NDIA  Systems 
Engineering  Division 

Select  organizations  that  develop  and  produce  products  (rather 
than  services) 

Identify  “focal”  person  within  each  organization 

•  Involved  with  /  interested  in  SE 

•  As  high  as  possible  within  the  organization’s  management  hierarchy 

Contact  Focals 

•  Brief  the  survey  and  solicit  their  support  within  their  organization 

•  Ask  them  to  solicit  respondents,  and  provide  the  tools  to  assist  them 

-  Respondent  solicitation  by  proxy  enhances  anonymity 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


The  Rigor - 

SEEC  Survey  Process 


October  21,  2008 


The  Rigor  —  Survey  Methodology 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey 

Population 

Organizations  developing  products  in  support  of  government 
contracts  (prime  or  subcontractors). 

Sampling  Method 

Invitation  to  qualifying  active  members  of  NDIA  Systems 

Engineering  Division.  Random  sampling  within  organization. 

Survey 

Deployment 

Web  deployment  (open  August  10,  2006  -  November  30,  2006). 
Anonymous  response.  Questions  based  on  CMMI-SE/SW/IPPD  vl.1 

Target 

Respondent 

Program  Manager  or  designee(s)  from  individual  projects 

Questionnaire 

Structure 

1 .  Characterization  of  the  project  /program  under  consideration 

2.  Evidence  of  Systems  Engineering  Best  Practices 

3.  Project  /  Program  Performance  Metrics 

Target  Response 
Time 

30  -  60  minutes 

Responses 

64  survey  responses  (46  complete;  18  partial,  but  usable) 

Analysis 

Raw  data  analyzed  by  Software  Engineering  Institute. 

Analysis  results  reviewed  by  NDIASE  Effectiveness  Committee. 

Reports 

1.  Public  NDIA/SEI  report  released  November  2007. 

2.  Restricted  attachment,  details  provided  to  respondents  only. 
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The  Rigor 

Analysis 


Perf-  f  (PC,  PE ,  SEC ,  AC) 

where:  Perf=  Project  Performance  PC  =  Project  Challenge 

PE  =  Project  Environment  AC  =  Acquirer  Capability 

SEC  =  Systems  Engineering  Capability 

SEC  can  be  further  decomposed  as: 

•  Project  Planning 

•  Project  Monitoring  and  Control 

•  Risk  Management 

•  Requirements  Development  and  Management 

•  Technical  Solution 

-  Trade  Studies 

-  Product  Architecture 

•  Product  Integration 

•  Verification 

•  Validation 

•  Configuration  Management 

•  IPT-Based  Capability 

SE  capabilities  and  analyses  are  fully  defined  by  mappings  of 
associated  survey  question  responses 
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The  Rigor  -  Terminology  and  Notation 
Distribution  Graph 


STRENGTH  T1IK01G1I  INDUSTRY  &  TECHNOLOGY 


Data 

Range 


(responses  to  corresponding 
survey  questions) 
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STRENGTH  T1IK01G1I  INDUSTRY  &  TECHNOLOGY 


The  Rigor  - 

Validation  of  Survey  Responses 


Maximum  =  3.9 
3rd  Quartile  =  3.3 
Median  =  3.0 
1st  Quartile  =  2.7 
Minimum  =  2.1 
N  =  63 


Overall  SE  Capability  (SEC) 


Maximum  =  4.0 
3rd  Quartile  =  3.1 
Median  =  2.S 
1 st  Quartile  =  2.4 
Minimum  =  1 .5 
N=  64 


Acquirer  Capability  (AC) 


12  3  4 


Maximum  =  4.0 
3rd  Quartile  =  3.1 
Median  =  2.75 
1 Quartile  =  2.3 
Minimum  =  1 .7 
N=  46 


Project  Performance  (Perf) 


Analyzed  distributions,  variability,  relationships... 
To  ensure  statistical  rigor  and  relevance 
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Variable  A 


Analysis 

MOSAIC  Charts  i 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


•High 


•Med 


•Low 


•A=High 

•A=High 

•A=High 

•B=Low 

•B=Med 

•B=High 

•A=Med 

•A=Med 

•A=Med 

•B=Low 

•B=Med 

•B=High 

•A=Low 

•A=Low 

•A=Low 

•B=Low 

•B=Med 

•B=High 

•Low 

•Med 

•High 

•Variable  B 
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The  Results! 

Mosaic  Chart 


Terminology  and  Notation  ^ 


Column  width 


represents  proportion 
of  projects  with 
this  level  of  capability 


Relative  performance 
distribution  of  the 
sample 


Gamma:  measures  strength  of 
relationship  between  two  ordinal 
variables 

probability  that  an  associative 
relationship  would  be  observed 
by  chance  alone 
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The  Results!  —  Total  SE  Capability 

(SEC)  vs .  Project  Performance  (Perf) 


STRENGTH  T1IK01G1I  INDUSTRY  &  TECHNOLOGY 


1 .00  - 


0.75- 


0.50- 


0.25  - 


0.00 


15% 

12% 

46% 

59% 

39% 

29% 

Lower 

Moderate 

Capability 

Capability 

(x  £2.5) 

(2.5<x<3.0) 

N  =  13 

N  =  17 

13% 


Higher 
Capability 
(x  £3.0) 
N  =  16 


Best 

Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x  £  3.0) 

Lower 

Performance 
(x  <  2.5) 


Gamma  =  0.32 
p  =  0.04 


Projects  with  better  Systems  Engineering  Capabilities  deliver  better 
Project  Performance  (cost,  schedule,  functionality) 
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The  Results!  -  Higher  SE  Capabilities 

are  Related  to  Better  Program  Performance 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


■  _ 

1.  Product  Architecture 


2.  Trade  Studies 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(xs  2.7)  (2.7<  x  <  3.3)  (x*3.3) 

N  =  1 8  N  =  1 4  N  =  1  3 


Gamma  =  0.40 
p  =  0.002 


Best  Performance 
(x  >  3.0) 


Moderate 
Performance 
(2.5  <  x  ^  3.0) 


Lower  Performance 
(x<2.5) 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(xs  2.7)  (2.7  <  x  <  3.3)  (xs  3.3)  Gamma  =  0.37 

N=  18  N  =  1 2  N  =  16  p  =  0.03 


3.  Technical  Solution 

4.  IPT  Capability 

5.  Requirements 
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The  Results!  ■  Relating  Project  Performance  mb^b^ 
to  Project  Challenge  and  SE  Capability  BSKIES 


Project  challenge 
factors: 

•Life  cycle  phases 
•Project  characteristics 
(e.g.,  size,  effort, 
duration,  volatility) 
•Technical  complexity 
•Teaming  relationships 


Projects  with  better  Systems  Engineering  Capabilities  are  better 
able  to  overcome  challenging  environments 
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The  Results!  - 

Summary  of  Process  Relationships 
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-Q 

<S 

Q. 

cts 

O 

LU 

CO 


Relationship  of  SE  Processes  to  Program  Performance 


Architecture 
T rade  Studies 
Technical  Solution 
IPT  Capability 
Reqts  Devel  &  Mgmt 
Validation 
Risk  Mgmt 
Verification 
Product  Integration 
Config  Mgmt 
Project  Planning 
Project  Monitor/Control 


-13%  C 


137 


40% 

p/o 


1 36% 


1134% 


123% 


123% 


3  25% 


3 1 3% 


:i3% 


]21% 


3  33% 


-20%  -10%  0%  10%  20%  30% 

Gamma  (strength  of  relationship) 


40% 


50% 


^  Strong  Relationship 


I  I  Moderately  Strong 
' — '  to  Strong  Relationship 


I  I  Moderately  Strong 
' — '  Relationship 


^  Weak  Relationship 
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Value  of  the  Research 


STRENGTH  T1IKOI  G1I  INDUSTRY  &  TECHNOLOGY 


Provide  guidance  for  defense  contractors  in  planning 
capability  improvement  efforts 

Establish  an  SE  Capability  Benchmark  for  defense  contractors 

Provide  justification  and  defense  of  defense  contractor  SE 

investments 

Provide  guidance  for  acquirer  evaluations  and  source 
selections 

Provide  guidance  for  contract  monitoring 

Provide  recommendations  to  OSD  for  areas  to  prioritize  SE 
revitalization 
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Conclusions  &  Caveats  - 

Summary 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


SE  Effectiveness 

•  Provides  credible  measured  evidence  about  the  value  of 
disciplined  Systems  Engineering 

•  Affects  success  of  systems-development  projects 

Specific  Systems  Engineering  Best  Practices 

•  Highest  relationships  to  activities  on  the  “left  side  of  SE  Vee” 

•  The  environment  (Project  Challenge)  affects  performance  too: 

-Some  projects  are  more  challenging  than  others  ...  and  higher 
challenge  affects  performance  negatively  in  spite  of  better  SE 
-Yet  good  SE  practices  remain  crucial  for  both  high  and  low 
challenge  projects 
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Conclusions  &  Caveats  - 

Next  Steps 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


*  Correlate  Report  Findings  with  Other  Sources 

•  Correlate  report  findings  with  results  of  OSD  systemic  root  cause 
analysis  project  (SEEC/OSD  work  group  established) 

*  Pursue  Specific  Improvement  Recommendations 
with  OSD 

•  Policy,  Compliance,  Education,  Data  Collection  (specific 
recommendations  submitted  to  OSD) 

*  Conduct  Additional  Analysis  of  Collected  Data 

•  Independent  Verification  &  Validation 

•  Discover  other  relationships  and  correlations 

*  Expand  the  Survey  to  Gauge  Improvements 

•  Incorporate  Lessons  Learned  from  participants 

*Expand  the  Survey  to  Commercial  Industries 

•  Discussion  with  IEEE  AEES  Board  of  Governors 

*  Survey  Acquirers 
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Survey  Results 


STRENGTH  T1IK01G1I  INDUSTRY  &  TECHNOLOGY 


“A  Survey  of  Systems  Engineering  Effectiveness-Initial  Results ” 
(CMU/SEI-2007-SR-014)  available  for  download  as  a  PDF  file  on  the 
SEI  web  site  at: 

http://www.sei.cmu.edu/publications/documents/07.reports/07sr014.html 
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SE  Effectiveness 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Questions? 


19[h  Annua!  International  Symposium  of  INCOSE  .  3  rd  Asia-Pacific  Conference  on  Systems  Engineering 

East  Meets  West 

INCOSE  H 

2009  jjjc 

The  Human  Dimension  to  Systems  Engineering 

SINGAPORE  (| 

Hosted  by  the  Region  VI  Chapters  of  Australia,  Beijing  Japan,  Korea,  Singapore  and  Taiwan 

20  -  23  |uly  2009 

Ken  Ptack 

ken.ptack(ci)incose.  ora 
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Back  -  up 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 
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DoD  Systemic  Root  Cause  Analysis 

-  Why  do  projects  fail? 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Insufficient  requirements  analysis  and  definition  at  program  initiation 

-  Not  tangible,  measurable,  testable,  stable 

-  User  R&M  requirements  are  not  underpinned  by  sound  rationale 

Acquisition  strategies  based  on  poor  technical  assumptions, 
competing  budget  priorities,  and  unrealistic  expectations 

Budget  not  properly  phased 
Lack  of  rigorous  systems  engineering  approach 

Schedule  realism  -  success  oriented,  concurrent,  poor  estimation 
and/or  planning 

Inadequate  test  planning  -  breadth,  depth,  resources 

Optimistic/realistic  reliability  growth  -  not  a  priority  during 
development 

Inadequate  software  architectures,  design/development  discipline, 
and  organizational  competencies 

Sustainment/life-cycle  costs  not  fully  considered  (short-sighted) 

SYSTEMS*.  SOFTWARE  ENGINE  ERIN  G  -  e- 2007  Page  5  O 


Top  10  Emerging  Systemic  Issues 

(from  52  "Deep  Dive" Program  Reviews  since  Mar  04) 


1 .  Management 

■  IPT  roles,  responsibilities,  authority,  poor  communication 

■  Inexperienced  staff,  lack  oftechnical  expertise 

2.  Requirements 

■  Creep/stability 

■  Tangible,  measurable,  testable 

3.  Systems  Engineering 

•  Lack  of  a  rigorous  approach,  technical  expertise 
■  Process  compliance 

4.  Staffing 

■  Inadequate  Government  program  office  staff 

5.  Reliability 

■  Ambilious  growth  curves,  unrealistic  requirements 

■  1  n  a  de  quate  "test  time "  fo  r  statisti  ca  1  ca  1  cu  lati  ons 

6.  Acquisition  Strategy 

■  C  ompeti  ng  bu  dg  et  p  ri  orities ,  s  c  he  du  le-  drive  n 

■  C  ontra  cting  issu  es ,  po  or  te  c  hn  i  ca  1  assum  pli  ons 

7.  Schedule 

■  Realism,  compression 

8.  Test  Planning 

■  Breadth,  depth,  resources 

9.  Software 

■  Architecture,  design/development  discipline 
•  Staffing/skill  levels,  organizational  competency  (process) 

1 0.  Maintainability/Logistics 

■  S  ustai  nment  c  osts  not  fu  1  ly  c  ons  id  e  re  d  (sho  rt-s  ig  hte  d) 

■  Supportability  considerations  traded 

Major  contributors  to  poor  program  performance 


SYS7EUS  *  SOFTWARE  ENGINEERING- 9.  2007 


... We  Don't  Manage  Them  Right 


•  I  nsuffi  ci  ent  trade  sp  ace 

-  Resources,  schedule,  performance,  requirements 

•  Insufficient  risk  management 

•  Inadequate  IMP,  IMS,  EVMS 

•  Most  programs  lack  quantifiable  entrance/exit  criteria 

•  Maturing  "suitability"  (e.g.,  RAM)  is  not  always  a  priority 

•  Maturing  "effectiveness"  is  not  always  a  priority 

•  Concurrent  test  program;  inadequate  scope  due  to  schedule  and 
resource  insufficiencies,  etc. 

•  Inadequate  OTRR  process  -  no  strong  DT&E  gate  prior  to  IOT&E 

•  Inadequate  government  staff;  Inexperienced  and/or  limited 
contractor  staffing 

•  Poorly  defined  IPT  roles,  responsibilities  and  authority 

-  Overall  poor  communications  across  government  and  industry  staff 

SYSTEMS*  SOFTWARE  ENGINEERS G -  QgMfiM  ».  2007  page 


Root  causes  from  DoD  analysis  of 

program  performance  issues  appear 
consistent  with  NDIA  SE  survey 
findings. 

Reference: 

Systemic  Root  Cause  Analysis, 

Dave  Castellano,  Deputy  Director  Assessments  & 
Support,  OUSD(A&T) 

NDIA  Systems  Engineering  Conference,  2007 
and.  NDIA  SE  Division  Annual  Planning  Meeting 
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Recommendations 


STRENGTH  T1IKOI  G1I  INDUSTRY  &  TECHNOLOGY 


1.  Policy:  Develop  policy  requiring  programs  to  apply  SE 
practices  known  to  contribute  to  improved  project 
performance. 

Contractual  compliance  to  bidder’s  SE  processes 

2.  Compliance:  Ensure  that  SE  practices  and  associated  work 
products  are  applied  to  projects  as  promised  and  contracted. 

Verification  via  evaluations,  audits,  milestones,  reviews 

3.  Education:  Train  program  staff  in  the  value  and  importance  of 
SE  and  in  the  application  of  SE  policy. 

Including  SE  value,  policy,  technical  evaluation 

4.  Data  Collection:  Establish  means  to  continue  data  collection 
on  the  effectiveness  of  SE  to  enable  continuous  process 
improvement. 

Follow-on  surveys,  analysis,  trending 
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•Conclusions  &  Caveats  - 

Consistent  with  “Top  10  Reasons  Projects  Fail*” 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  Lack  of  user  involvement 

2.  Changing  requirements 

3.  Inadequate  Specifications 

4.  Unrealistic  project  estimates 

5.  Poor  project  management 

6.  Management  change  control 

7.  Inexperienced  personnel 

8.  Expectations  not  properly  set 

9.  Subcontractor  failure 

10. Poor  architectural  design 

*  Project  Management  Institute  Matching  items  noted  in  RED 

Above  Items  Can  Cause  Overall 
Program  Cost  and  Schedule  to  Overrun 
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■ 


Conclusions  &  Caveats  - 

Consistent  with  “Top  5  SE  Issues*”  (2006) 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


•  Key  systems  engineering  practices  known  to  be  effective  are  not  consistently 
applied  across  all  phases  of  the  program  life  cycle. 

•  Insufficient  systems  engineering  is  applied  early  in  the  program  life  cycle, 
compromising  the  foundation  for  initial  requirements  and  architecture 
development. 

•  Requirements  are  not  always  well-managed,  including  the  effective 
translation  from  capabilities  statements  into  executable  requirements  to 
achieve  successful  acquisition  programs. 

•  The  quantity  and  quality  of  systems  engineering  expertise  is  insufficient  to 
meet  the  demands  of  the  government  and  the  defense  industry. 

•  Collaborative  environments,  including  SE  tools,  are  inadequate  to  effectively 
execute  SE  at  the  joint  capability,  system  of  systems,  and  system  levels. 

*  OUSD  AT&L  Summit  Matching  items  noted  in  RED 
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The  Results!  - 

Summary  of  Relationships 


Driving  Factor 

Relationship  to  Project 
Performance 

Description 

r 

Requirements  and 
Technical 

Solution  Combined 
with  Project  Challenge 

Very  strong  positive 

+0.63 

Combined 

Requirements  and 
Technical  Solution 

Strong  positive 

+0.49 

Product  Architecture 

Moderately  strong 
to  strong  positive 

+0.40 

Trade  Studies 

Moderately  strong 
to  strong  positive 

+0.37 

IPT-Related  Capability 

Moderately  strong 
positive 

+0.34 

Technical  Solution 

Moderately  strong 
positive 

+0.36 

Requirements 
Development 
and  Management 

Moderately  strong 
positive 

+0.33 
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Driving  Factor 

Relationship  to  Project 
Performance 

Description 

r 

Total  Systems 
Engineering  Capability 

Moderately  strong 
positive 

+0.32 

Project  Challenge 

Moderately  strong 
negative 

-0.31 

Validation 

Moderately  strong 
positive 

+0.28 

Risk  Management 

Moderately  strong 
positive 

+0.28 

Verification 

Moderately  strong 
positive 

+0.25 

Product  Integration 

Weak  positive 

+0.21 

Project  Planning 

Weak  positive 

+0.13 

Configuration 

Management 

Weak  positive 

+0.13 

Process  Improvement 

Weak  positive 

+0.05 

Project  Monitoring  and 
Control 

Weak  negative 

-0.13 
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Agenda 

LOCKHEED  MART 

We  never  forget  who  we 7  re  working  for™ 

INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 

■  Social  Concerns 

■  Paradigm  Shift 

■  360-Degree  View 

■  SE  Leadership  Theory 

■  Thompson’s  Alignment  Model 

■  Success  Story 

■  Emerging  Alignment  Themes 

■  Conclusion 

■  Q  &  A 


Social  concerns  and  theoretical  interests 


•  Lack  of  understanding  IT’s  business  value 

•  Ever  changing  organizational  structure 

•  Danger  of  IT  overspending 

•  Increasing  IT  spending 

•  Increasing  dependence  on  IT 

•  The  changing  CIO  roles 

•  IT  and  business  alignment  is  a  must 

•  The  pressing  urgency 

•  Establish  irreversible  momentum  for  change 


Thompson  (2008) 


What  is  a  Paradigm  Shift' 

INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


■  Thomas  Kuhn  (1962)  first  used 
this  term  in  his  influential  book, 
“The  Structure  of  Scientific 
Revolutions”,  to  describe  a 
change  in  basic  assumptions 
within  the  ruling  theory  of 
science. 

■  Jastrow  (1899)  used  the  duck- 
rabbit  optical  illusion  to 
demonstrate  the  way  in  which  a 
paradigm  shift  could  cause  one 
to  see  the  same  information  in 

an  entirely  different  wav. 

■  The  term  has  been  adopted 
since  the  1960s  and  applied  in 
non-scientific  contexts 
(Wikipedia) 


LOCKHEED  MARTIN 

We  never  forget  who  we 1 re  working  for™ 


The  famous  duck-rabbit  ambiguous  image.  Is 
it  a  duck?  Is  it  a  rabbit? 

Source:  Jastrow,  J.  (1899).  The  mind's  eye. 
Popular  Science  Monthly,  54,  299-312. 
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Paradigm  Shift  for  SE 

Professionals 

INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


I  ndividual  Contributor 


LOCKHEED  MARTIN 

We  never  forget  who  we 1 re  working  for™ 


View,  Understand,  Map,  &  Manage 


Program  Leadership 
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Paradigm  Shift  for  SE  Loaders 

LOCKHEED  MARTIN 

We  never  forget  who  we 1 re  working  for™  * 

INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 

■  360-Degree  Leader  Program  Leadership 

■  Serves  others 

■  Needs  to  practice  and  be  trained 

■  Works  as  a  program  leader 

■  Shines  as  a  setting  sun:  Make  others  successful 

■  Strategy  &  Business  Leader 

■  Encourages  Teamwork 

■  Works  as  a  Coordinator 

■  Makes  wise  decisions 

■  Works  as  a  project  leader 

■  Has  risk  of  losing  passion  of  technical  leadership 

■  Shines  as  a  high  noon:  Strong 

■  Technology  Leader 

■  Is  a  leader  of  technology 

■  Is  a  hero/heroine  for  warriors 

■  Works  as  a  technical  task  leader 

■  Has  risk  of  asking  too  much  of  a  control 

■  Shines  as  a  rising  sun:  Potential 

I  ndividual  Contributor 


360-Degree  View  for  SE  Leaders 


LOCKHEED  MARTIN 

We  never  forget  who  we  *re  working  for™ 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


■  Leadership 

■  Visionary:  Provide  vision  for  changes 

•  Core  values  (what  we  stand  for,  that  is,  Imagination:  Walt 
Disney) 

•  Core  purpose  (why  we  exist,  that  is,  To  make  people  happy: 
Walt  Disney) 

•  Envisioned  future  includes  long-term  goals  (that  is,  Become 
the  Harvard  of  the  West:  Stanford  University,  1940s) 

■  Technical 

■  Business 

■  Functional 

■  Managerial:  Produce  plans  for  stability  and  leaders 

■  Technology 

■  Process 

■  People 


Boehm  &  Ross’  (1989)  Leadership 


LOCKHEED  MARTIN 

We  never  forget  who  we  Jre  working  for™ 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


■  Theory  W  360- Degree  Leader 

■  Negotiator 

■  B.W.  Boehm  and  R.  Ross, 1989 

■  Make  everyone  a  winner 

■  Theory  Z 

■  Facilitator 

■  Motivation  and  Productivity  (Gellerman,  1978) 

■  Do  up-front  investment  in  developing  shared  values  and  arriving  at  major 
decisions  by  consensus  within  an  organization  strat  &  Business  Leader 

■  Theory  Y 

■  Coach 

■  Productive  Software  Management  (Evans,  Piazza,  &  Dolkas,  1983) 

■  Stimulate  creativity  and  individual  initiative 

■  Theory  X  Technology  Leader 

■  Autocrat 

■  Scientific  Management  (Taylor,  1911) 

■  Do  more  precise  time  and  motion  studies 

■  Organize  jobs  into  well-orchestrated  sequences  of  tasks 


Boehm,  B.  W.,  &  Ross,  R.  (1989).  Theory-w  software  project  management:  Principles  and 
examples.  IEEE  Transactions  on  Software  Engineering,  15(7),  902-916. 


Theory 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


LOCKHEED  MARTIN 

We  never  forget  who  we  're  working  for™ 


■  Builder  of  Learning  Organizations 

•  Here  is  our  purpose  and  direction  -  I  will  guide  and 
coach! 

■  Group  Facilitator 

•You  are  empowered! 

■  Task  Manager 

•  Here  is  what  to  do  and  how  to  do  it! 

■  Bureaucratic  Manager 

•  Follow  the  rules! 


The  role  of  leadership  in  software  development  by  Mary  Poppendieck,  2007  (Originally  from  The 
Toyota  Way,  Jeffery  Like,  p.  181) 


LM  Full  Spectrum  Leadership 

LOCKHEED  MARTIN ^ 

We  never  forget  who  we 1 re  working  for™  * 

INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 

Full  Spectrum  Leadership 

The  Language  of  Leadership  at  Lockheed  Martin 


■Shape  the  Future 

■Build  Effective 
Relationships 

■Energize  the  Team 

■Deliver  Results 

■Model  Personal 
Excellence,  Integrity, 
and  Accountability 
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Thompson’s  (2000)  / 

Mignm 

tent  Model 

LOCKHEED  MARTIN 

We  never  forget  who  we 7  re  working  for™  * 

INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 

Business  Performance 


“OS0 


Strategic 

Management 


Enterprise  Management 
People 


Organization 


Capability 

Production 

Management 


Process 


Relationship 

Management 


©-structure 


Operations  & 
Maintenance 
Management 


Success  Story 

LOCKHEED  MARTIN 

We  never  forget  who  we  Jre  working  for™  * 

INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 

■Program  Overview: 

■  Provides  a  wide  range  of  systems 
engineering  services  to  a  civilian 
government  agency  nationwide 

■  Nine-year  contract  worth  approximately 
$700  million 

■Indefinite  Delivery/Indefinite  Quantity 
(IDIQ) 


Initiative  1;  Unified  Development 
Environment  -  TECHNOLOGY 


LOCKHEED  MARTIN 

We  never  forget  who  we  *re  working  for™ 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


■Restructured  and  empowered  to  implement  the  program¬ 
wide  technology  governance  and  sharing 

■  Architectural  Control  Board  (ACB) 

■  Organizational  Process  Group  (OPG) 

■  Sr.  Technical  Council 

■Established 

■  Chief  Technology  Officer  (CTO)  360-Degree  Dashboard 

■  Technology  Inventory 

■  Distributed  Software  Development  Team  (Develop  globally, 
manage  centrally) 

■  Continuous  Integration  &  Automated  Testing 

■  Standard  Defect  Tracking 

■  Document  and  Knowledge  Management 

■  Removing  Accidental  Complexity  from  Architectures 

■  Challenge  -  Action  -  Results 


Initiative  2;  Technology  Vision  and 
Roadmap  -  STRATEGY 


LOCKHEED  MARTIN 

We  never  forget  who  we  *re  working  for™ 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


■Collaborate  with 

■  Customer 

•  Enterprise  Architecture  (EA)  Workgroup 

•  Web  Workgroup 

•  Portal  Workgroup 

•  SOA  Workgroup 

•  GIS  Workgroup 

•  National  Computer  Center 

■  Industry 

•  Software  Vendors 

•  Consortia 

■  LM 

•  LM  Engineering  Process  Improvement  Group 

•  LM  Center  of  Excellence  (COE) 

•  LM  IS&GS  Advanced  Technology  Group 

•  LM  NexGen 

•  LM  l&KS  Technical  Council 


Initiative  3: 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


LOCKHEED  MARTIN 

We  never  forget  who  we 7  re  working  for™ 


■  Provide  the  active  and  quality  support  to 
the  Task  Order  Project  Officers  (TOPO) 
and  Contract  Technical  Managers  (CTM) 
to  solve  their  business  challenges  in  a 
timely  fashion. 

■  Conduct  the  analysis  of  customer  needs 
to  ensure  the  program  provides  the 
leading-edge  solutions  that  meet  and 
exceed  customer  expectations. 

■  Restructure  one  of  Task  Orders  to  include 
consultations  on  the  Enterprise  Tools 

Best  Practices. 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


LOCKHEED  MARTIN 

We  never  forget  who  we 7  re  working  for™ 


■  Establish  a  viable  business  operation  model 

■  Earn  trust,  enhance  competency,  and  establish  strategic 
partnerships 

■  Pursue  IT  capability  as  a  means  of  enhancing  business  functions 

■  Expand  skills,  build  teams,  and  maximize  productivity 

■  Instill  an  effective  governance  structure 


(Thompson,  2008) 
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Conclusion 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


LOCKHEED  MARTIN 

We  never  forget  who  we 7  re  working  for™ 


■360-Degree  View  is  proven  to  be 
necessary  and  helpful  for  further 
aligning  business  and  technology 

■Business  management  aligned  with 
technology  planning  often  enhances 

business  performance  (Thompson,  2008) 


Thompson,  S.  (2008).  A  Qualitative  Study  of  Successful  Practices  in  Aligning  Information  technology  and  Business  Management 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


LOCKHEED  MARTEN 

We  never  forget  who  we  ’re  working  for™ 


Questions? 


Contact  Information 


INFORMATION  SYSTEMS  &  GLOBAL  SERVICES 


Min-Gu  Lee 

Chief  Architect 

Lockheed  Martin  ITS-ESE  Program 

Chief  Technology  Officer 

Lockheed  Martin  Environmental  &  Technical 
Service  Line  of  Business 

Telephone:  703-647-5830 

E-mail:  mi n-qu.leeCcDlmco.com 


LOCKHEED  MARTIN 

We  never  forget  who  we 1 re  working  for™ 


Dr.  Shue-J  ane  L.  Thompson 


Director,  Solution  Strategies 

Lockheed  Martin  Enterprise  Solutions  & 

Services 

Telephone:  703-389-9272 
E-mail:  shue-iane.thompsonCcDlmco.com 
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Outline 


Raytheon 

Integrated  Defense  Systems 


■  Cases  for  Action 

■  Health  Management  Enterprise  Architecture 

■  Prognostics  Systems  in  the  PHM  Enterprise 

-  Prognostics  Design  &  Development 

-  Prognostics  &  Health  Management  Concept 

■  Total  Asset  Visibility  Systems  in  the  PHM  Enterprise 

-  Total  Asset  Visibility  Concept 

-  Example  Mesh  Network 

■  Health  Management  Enterprise  Information  Flow 

■  Communications  Architecture  Considerations 

■  Role  of  Logistics  Planning  in  Mission  Planning 

■  Borrowing  from  Semantic  Web  Concepts 

■  Conclusion 


Cases  for  Action 


Raytheon 

Integrated  Defense  Systems 


■  Customers  are  demanding  Prognostics  &  Health  Management  solutions 
for  extending  product  life. 

■  Test  costs  are  rising  due  to  complex  design  and  test  requirements. 

■  In  the  short  run,  missions  can  fail  due  to  unpredicted  failures. 

■  In  the  long  run,  system  performance  is  not  well  maintained. 

■  We  can  guarantee  system  performance  and  lower  maintenance  by 
predicting  failures  before  they  occur. 

-  These  strategies  require  Prognostics  &  Health  Management  Technologies  and 
an  overall  Condition  Based  Maintenance  strategy. 


Health  Management  Enterprise 
Architecture 


Raytheon 

Integrated  Defense  Systems 


Mission 


Prognostics  Systems  in  the  PHM 
Enterprise 


Raytheon 

Integrated  Defense  Systems 


Mission 


Sensors 


Prognostics  Design  & 
Development 


Raytheon 

Integrated  Defense  Systems 


Prognostics  &  Health 
Management  Concept 


Raytheon 

Integrated  Defense  Systems 


Purpose:  By  predicting  system  Remaining  Useful 
Life,  we  can  remedy  failures  before  they  occur. 

Health  Assessment  step:  Determine  current 
state. 

-  e.g.  Fuel  levels  are  low  ->  Fuel  is  urgently  needed. 

Prognostics  Assessment:  Project  future  state. 

-  e.g.  Ship  radar  will  fail  in  the  next  72  -  96  hours. — 
Order  a  replacement  part  immediately. 

Advisory  Generation:  Recommend  maintenance 
strategy  based  on  overall  system  or  fleet  health. 


Advisory  Generation 


t 


Prognostics  Assessment 


t 


Health  Assessment 


t 


State  Detection 


Data  Manipulation 


t 


Data  Acquisition 


t 


Sensor 


These  functions  can  be  performed  on  or  off- 
board  the  platform  of  interest. 


Total  Asset  Visibility  Systems  in 
the  PHM  Enterprise 


Raytheon 

Integrated  Defense  Systems 


Mission 


Total  Asset  Visibility  .  ;  ™"y0nn9d 
Concept  ooct  rf® 


Raytheon 

Integrated  Defense  Systems 


We  are  developing  technologies: 


Wireless  Sensor  Nodes 


■  Which  enable  nodes  to  report  status  by  forwarding 
data  through  a  mesh  network. 


■  That  allow  assets  to  be  tracked  throughout  their 
lifecycle — not  just  during  shipment. 

-  This  allows  us  to  track  when  and  where  failures  occur. 

-  Better  failure  diagnosis  and  prognosis  becomes  possible. 


Future 

Evolution 


For  the  Future: 

■  We  are  miniaturizing  Wireless  Sensor  Nodes  for 
embedding  into  platforms.  (See  Terry  Tracy’s 
MILCOM  paper) 

■  To  make  robust  Wireless  Sensor  networks,  we  are 
researching  Disruption  Tolerant  Networking  schemes. 


Wireless 

Communication 


Rowe 


Seif  Adhesive 


Health  Management  Enterprise 
Information  Flow 


Raytheon 

Integrated  Defense  Systems 


Maintainer 


repairs 


searches 
parts  dbase 
and  repair 
strategies. 


(2) 

Health 
alerts  sent 
to  logistics 
planners  \ 


Logistics 

Planner 


(3) 

Commander 

re-plans 

mission 


(2) 

Health  alerts 
sent  to 
mission 
planners 

(i) 

Health  data 

collected 

and 

analyzed 


Mission 


Role  of  Logistics  Planning  in 
Mission  Planning 


Raytheon 

Integrated  Defense  Systems 


Background:  The  program  manager  determines  a  maintenance  strategy  and  schedule 
based  on  how  his  fleet  will  be  employed. 


Fleet  deployed  into  new, 


hostile  environment 


Equipment  is  exposed  to 
extreme  heat,  terrain,  etc. 


► 


Commander  predicts 
higher  usage  of  fleet 


Logistics  PM  determines 

mission  impact  based  on 
mission  employment  and 
environmental  stresses 


Logistics  PM  forecasts 
equipment  degradation 
within  2  months 

\ 

Logistics  PM  replans 
sustainment  strategy 


Borrowing  from  Semantic  Web 
Concepts 


Raytheon 

Integrated  Defense  Systems 


■  To  enable  fast  and  automated  failure  response,  we  need: 

-  The  ability  to  organize  and  aggregate  large  quantities  of  information  so  that 
they  can  be  analyzed. 

-  Interoperability  via  a  common  language  framework.  -  Key 

■  An  example  of  the  future: 

-  Tom,  a  logistics  planner,  receives  an  alert  about  a  potential  failure. 

-  His  planning  tool  auto  generates  a  list  of  repair  strategies,  with  associated  info 
about  cost,  schedule  impacts,  historical  effectiveness,  and  resource  needs. 

-  The  tool  recommends  a  strategy  providing  the  quickest  repair. 

-  Tom  doesn’t  like  this  choice,  since  it  involves  some  risk  of  unsuccessful  repair. 

-  Tom  requests  another  option  and  inputs  detailed  requirements  and  goals. 

-  The  planning  tool  returns  a  recommendation  for  a  more  failsafe  approach, 
which  requires  additional  materials  and  changes  to  the  repair  schedule. 

-  Tom  selects  this  option  and  approves  ordering  of  the  needed  materials. 


Conclusion 


Raytheon 

Integrated  Defense  Systems 


■  Raytheon  is  tackling  the  Mission  Support  problem  space  from  a  System 
of  Systems  approach. 

■  Through  a  DoDAF  architecting  process,  we  seek  to  understand  key 
warfighter  needs. 

■  We  are  modeling  the  architecture  from  a  total  system  view,  to  integrate 
core  PHM  products  into  an  end-to-end  PHM  solution. 

-  Sensors  and  Prognostics  algorithms  to  build  equipment  health  status. 

-  Total  Asset  Visibility  to  provide  asset  location  and  general  status. 

-  Integrated  Information  Management  to  organize  the  most  relevant  health  status 
and  asset  information. 

■  Using  a  reference  PHM  Architecture,  we  can  quickly  deploy  concept 
demos  and  new  product  solutions. 

■  The  Prognostics  and  Health  Management  Enterprise  enables  us  to 
maintain  system  performance  for  the  long  run. 


11th  Annual 

NDIA  System  Engineering  Conference 

Enterprise  Health  Management  Committee 

Electronics  Prognostics  Technology  Study 
E-Prog  Figure  of  Merit  Application 


23  October  2008 


Briefing  Topics 


•  The  Background  of  NDIA  Electronic  Prognostics  Studies 

-  Why  Electronic  Prognostics 

-  The  Trail  to  The  Current  Application  Study 

-  NDIA  Study  Results 

-  Some  Electronic  Prognostics  Figures  of  Merit  (FOM) 

•  Putting  Numbers  on  the  Figures  of  Merit 

-  The  Process  for  FOM  Computation 

-  The  Results  -  Data,  Analysis,  Computation  of  FOM  Values 

•  Application  of  the  FOM  Results  to  the  Fleet 

-  Air  Force 

-  DOD 

•  Next  Steps 


2 


Why  Electronics  Prognostics 


•  Greater  reliance  on  electronics  and  electrical  based  systems: 

-  Navy  -  JSF,  EMALS,  AAG,  Shipboard  Weapons  Loader,  shipboard  electric 
drive,  Integrated  Fight  Through  Power,  ForceNet,  linear  motor  elevators,  etc. 

-  Army  -  FCS  Hybrid  electric  drive,  soldier  mounted  electronics,  MTRS,  Net 
Centric  Warfare,  etc. 

-  AF  -  JSF,  F-22 

•  Enables  users  ability  to  operate  and  maintain  increasingly  sophisticated 
weapon  systems 

-  Prognostics  provides  advanced  warning  of  deterioration  as  opposed  to 
reporting  failure 

-  Potential  to  reduce  downtime  for  unscheduled  maintenance  and  reduce  costly 
secondary  damage  associated  with  failures 

-  Supports  emerging  distance  support  initiative 

•  Required  technology  to  enable  PHM,  Performance  Based  Logistics,  and 
Sense  and  Respond 
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Legacy  VS  Prognostics  Health  Management  (PHM) 
Summary  of  Expectations 


Maintainability 

MFHB  CND 
MFHBME 
MFHBR 
MMH/FH 

Support  Equipment 

QTY 

Weight  (Lbs.) 
Volume  (cu  ft) 


PHM  Benefits 

79-82%  Improvement 
13-14%  Improvement 
3%  Improvement 
17-32%  Improvement 


Reduction  of 
6-10% 


Manpower 

QTY 


Reduction  of 
46-52% 


Logistics  Footprint 

Cl  7  Loads,  Tons 

Safety 

Mishap  Reduction 


Reduction  of 
2-17% 


Reduction  of 
14-38 


SGR 

SGR  (Initial/Sustained) 


10  to  14% 
Improvement 


Airframe/OML  Restoration 

Recurring  Cost 


$1.05B  -$7.87B 
Cost  Avoidance 
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The  Trail  To  The  Current  Application  Study 


NDIA  E-Prog  I 

Workshop 
(JSF  2004  Tasking) 
Data  to  Enable 
Fielded  Electronic 
Prognostics  on  JSF 


Data  List 

19  Items 
of 

Contextual 

And 

Operational  Data 


mpm 


NDIA  Study  Results 
Post  E-Prog  II  Workshop  Process 


•  All  Gov’t  Task  IPT 

•  Developer  and  User  Focus  -  not  S&T 

•  Defined  in  Real  Prognostic  Terms  Based  on  Repair  and  Logistics  Delay  Times 

•  Prognostic  Horizon  -  How  much  longer  will  it  work  before  failing? 

•  Confidence  factor  -  %  confidence  that  the  estimated  Horizon  is  right 

PROCESS 


•  SBIR/STTR 

•  BAA 
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HVIVTIU 

E-PROG  R&D  PROGRAM  EXAMPLE  1 


Prognostics  for  Power  Supplies  and  Converters 


Program  Rationale:  This  program  area  addresses  the  need  for  prognostics  for  all  types  of  electronic  power  supplies  and  power 
converters.  Sensed  parameters,  sensor  performance  characteristics,  sensor  configuration  (built  into  or  added  on  to  the  device), 
data  analysis  algorithms,  degree  of  smart  sensing  and  integration  with  other  electronic  and  electromechanical  prognostic 
technologies  are  all  a  part  of  this  effort.  The  Verification  and  Validation  of  the  prognostic  technology  are  included  as  part  of 
this  program. 

Key  Program  Elements: 

•  Implementable  prognostics  for  power  supplies/converter. 

•  Transition  of  current  SBIR  technology  to  wider  applications. 

•  Development  of  additional  technology  where  needed. 

•  Incorporate  in  new  designs  and  appended/integrated  in  current  designs 

Horizon:  T=100hr  Confidence:  T  =  90% 

0=  1000  hr  0=95% 


S  &  T  Category 

Estimated  Duration 

(Year  s) 

Budgetary  Man- 
Years 

6.1  Basic  Research 

0 

0 

6.2  Applied  Research 

2 

16 

6.3  Advanced  Technology  Development 

2 

16 

6.4  Advanced  Component  Development 

1 

S 

Total 

5 

40 

Table  20.  E-PROG  Program  19  Development  Plan 
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NDIA  Study  Results 

Program  Breakout  by  R&D  Category  and  Product  Type 


E-Prog  Description 

6.1 

Basic 

Research 

6.2 

Applied 

Research 

6.3 

Tech  Demo 

6.4 

Tech 

Application 

Prod. 

Type 

1 .  Physics  of  Failure  Model  for  Gates,  Devices  and  IC’s  « 

M 

2.  Electronics  Prognostics  for  High  Power  Switching 

PT 

Electronics 

r  i 

3.  BIP  Prognostics  for  Devices  and  Circuit  Boards 

PT 

4.  Electronics/electro-optical  Prognostics  for  Tactical 

PT 

Sensor  Systems 

5.  Generic  Environmental/Operational  Parameter 

Monitoring  Module  for  Electronic  Prognostics 

H 

6.  Electronic  Prognostics  for  C4ISR  Systems 

PT 

7.  Maintenance  Mode/Prognostic  Interaction  Design  Tool 

T 

8.  Interconnection  Prognostic  Technology 

PT 

9.  Electronic  Interconnection  Prognostic  Design  Tools 

T 

10.  Electronics  Prognostics  Financial  Modeling  Tool 

T 

11.  Tool  for  Logistics  Impact  of  E-Prog 

T 

12.  Prognostics  for  HCI  Electronics/Electro-Optics 

PT 

13.  Prognostics  for  Redundant  Electronic  Systems 

PT 

14.  Electronic  Prognostics  Design  Tool  for  Environmentally 

rri 

Tolerant  Electronics 

T 

15.  Electronics  Life  Usage  Assessment  and  Prognostics  - 

Electronic  Prognostics  Life  Usage  System  (E-Plus) 

PT 

16.  Data  Enterprise  System  -  Module  to  LRU  Tracking  for 

PT 

Electronics  Prognostics 

17.  Electronic  Prognostics  Reasoner  Engine  applicable  to 

PT 

Device  through  System 

18.  Electronic  System  Level  Prognostic  and  RUL  Tool  Set 

T 

19.  Prognostics  for  Power  Supplies  and  Converters 

PT 

M  =  Model,  H  =  Hardware,  PT  =  Prognostic  Technology,  T  =  Tool 
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NDIA  Study  Results 
Road  Map 


Man  Year  Summary  By  FY 


FY1 

FY2 

FY3 

FY4 

FY5 

FY6 

FY7 

FY8 

Total 

6.1  Total 

41 

45 

26 

112 

6.2  Total 

28 

48 

112 

80 

32 

300 

6.3  Total 

0 

20 

20 

72 

88 

60 

B 

26  B 

6.4  Total 

0 

0 

3 

16 

20 

44 

44 

8 

140 

Totals 

69 

113 

166 

168 

140 

104 

52 

8 

B20 

•Nearly  70%  of  Program  is  6.2  &  6.3  -  only  14%  of  Program  is  6.1 
•Benefits  of  effort  start  to  be  realized  in  FY3 
•Majority  of  effort  is  completed  within  4-5  years 
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Some  Electronic  Prognostics 
Figures  of  Merit 

Potential  Areas  Where  Electronics  Prognostics  Could  Offer 
Significant  Benefits  to  Advanced  Military  Systems 


Benefit  Area  FOM  Metric 


Total  cost  of  ownership  reduction 

%  Reduction  in  Support  Cost, 
Material  &  Labor 

Reduction  of  cost  of  false  removals 

%  Reduction  &  Cost  Savings  on 
Spares  &  NFF/RTOK 

Improved  system  availability 

%  Reduction  in  NFMC  and 
Recovered  Sorties 
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Putting  Numbers  on  the  Figures  of  Merit 

The  Process 


•  Select  a  Program  for  FOM  Analysis 

-  Fielded  Air  Force  Fixed  Wing  (F/W)  Aircraft 

-  High  Mission  Electronics  Content 

-  Analysis  of  50  Mb  Support  Data  from  Approximately  Wing  Size  Sample 

-  Analyzed  a  2  Year  Operational  Period,  Annualized  Results 

•  The  Analysis  Approach 

-  Calculate  the  Component  Parameter  Values 

•  Mission  Aborts  from  Electronic  Causes  -  Replacement  Weapon  Systems  to 
Reestablish  the  Mission  Rate 

•  MMH  for  Electronics  Maintenance  -  Reductions  from  Embedding  E-Prog 

•  Excess  Spares  Usage  and  Inventory  -  Due  to  lack  of  Embedded  E-Prog 

•  NFF/RTOK  Rate  -Material  and  Labor  Cost  due  to  lack  of  Embedded  E-Prog 

-  Assemble  the  Component  Parameter  Values  into  The  FOMs 


ll 


Putting  Numbers  on  the  Figures  of  Merit 

Analysis  of  Expected  Savings  From  Embedded  Electronic  Prognostics 


Calculated  Component  Parameter  Values 


•  Mission  Aborts  from  Electronic  Causes 

-  NMC  Aborted  Takeoffs  +  In-Uight  Aborted  Missions  =  55  (8%) 
2  Additional  A/C  per  Wing ) 

-  NFMC  Missions  (Prior  to  Takeoff  and  In  Flight)  =  335  (47%) 


•  NFF/RTOK  Rate  -  Related  Material  and  Labor  Cost 

-  Total  Maintenance  =  33,000  MMH 

-  Total  Electronic  Maintenance  =  5,300  MMH  (LRU  Replacement)  =  16%  of  Total  MMH 

-  NFF  and  FD/FI  =  4,630  MMH  (87%  of  Electronic  MMH  or  14%  of  Total  MMH) 

-  NFF  /  RTOK  Rate  14-22%  (18%Avg.)  =  Equivalent  of  4  Electronic  Systems  in  Pipeline 
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Putting  Numbers  on  the  Figures  of  Merit 

The  Results 


Component  Parameter  Values  Assembled  into  FOM 


Total  cost  of  ownership  reduction 
(Support  Cost  For  Example  W/S) 

Reduction  in  Support  Labor  =  14% 

Reduction  in  Electronic  Support  Material  =18% 

(4  electronic  Systems  per  Wing) 

Reduction  of  cost  of  false  removals 

Reduction  &  Cost  Savings  NFF/RTOK  =  14% 

Reduction  &  Cost  Savings  on  Spares  =  18% 

Improved  system  availability 

Reduction  in  NMC  =  8%  (or  2  A/C  per  Wing) 

Reduction  in  NFMC  =  47% 
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FOM  Results  Applied  to  the  FW  A/C  Fleet  (Est.) 


USAF  Tactical  FW  A/C  (2006) - 

DOD  Tactical  FW  A/C  (2006) - 

(From  2006  DOD  GAO  Study) 

Est.  Avg  Unit  Cost - 

Est.  Avg  Electronics  Content - 

DOD  Electronics  Maintainers  FW  A/C  Est. — 

DOD  Labor  Cost@$45KPer - 

USAF  is  30% - 

2500  Estimated  Corporate  Maintenance  Indicators  -  USAF 

3700  (From  2006  DOD  GAO  Study) 

$  ^8MM  Mission  Capable  Rate -  81% 

12,500  NMC-Maintenance -  15% 

$  560  Million  A , 

$170  Million  Abort  Rate  6/o 

Total  cost  of  ownership  reduction 

Reduction  in  Support  Labor  =  14%  =  $  46  Million  (USAF) 

Reduction  in  Electronic  Support  Material  =18%  =  $  101  Million  (USAF) 

Reduction  in  Support  Labor  =  14%  =  $  69  Million  (DOD) 

Reduction  in  Electronic  Support  Material  =18%  =  $  150  Million  (DOD) 

Reduction  of  cost  of  false  removals 

Reduction  &  Cost  Savings  on  Support  Material  =14%  =  $  46  Million  (USAF) 
Reduction  &  Cost  Savings  NFF/RTOK  =  18%=  $  101  Million  (USAF) 

Reduction  &  Cost  Savings  on  Support  Material  =14%  =  $  69  Million  (DOD) 
Reduction  &  Cost  Savings  NFF/RTOK  =  18%=  $  150  Million 

Improved  system  availability  (DOD 

Reduction  in  NMC  =  8%  =  $  8  Billion  (USAF) 

Reduction  in  NMC  =  8%  =  $  11.8  Billion  (DOD) 

14 

Recommended  Next  Steps 


mmrm 


•  Expand  Study  to  Classes  of  Weapon  Systems 

-  Select  Best  Payoff  Classes  (Troubled) 

-  Prescribe  Specific  E-Prog  Programs 

-  Develop  Specific  Cost  Benefit 

•  Develop  Programs  and  Acquisition  Strategy  for  the  Prescribed  E-Prog 
Technologies 

•  Execute  Programs  and  Develop  Technology  Transition  Plan 

•  Develop  Metrics  and  Evaluate  Results 

•  Repeat  for  Additional  classes  of  Weapon  Systems. 
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Topics 


Raytheon 


■  Disaster  awaits 

■  Mission  is  the  context  for  systems  engineering 

■  Mission  analysis  -  building  the  ‘right’  mission  knowledge 
foundation 

■  Tools  of  the  trade 
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Disaster  Awaits 


Raytheon 

Customer  Success  Is  Our  Mission 


Raytheon 


Hyatt  Regency  Walkway  Collapse  - 1981 


Interface  failures 
(connections) 


\ 


Moved  from 
concept  to 
design  too  * 

quickly 

Used  beyond 
design 
limitations 

Communication 

failures 


Critical  product 
components 
departed  from 
design  detail 

♦ 


117  people  died 
$140M  cost  (judgments 


alone) 

Negative  national  press 

t 

Key  product 
‘subsystems’ 
developed  by 
subcontractor 


Construction 
began  before 
design  was 
sufficiently 
mature 


Inspection  for  earlier 
significant  failure 
limited  due  to 
scope  issues 


Critical  changes 
accepted  via 
verbal 
approval 


Mission  Need  Exceeded  the  Capability 
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What’s  Similar  Between  a  Walkway  and  Raytheon 
a  Weapons  System? 

■  Mission  should  be  pre-eminent  in  our  planning  and  building 

■  Operational  use  will  expand  beyond  existing  design  capability 

■  Communication  too  often  lacks  clarity,  conciseness,  rigor 

■  Prime  hires  others  to  provide  piece  parts  for  the  solution 

■  Interfaces  are  high  risk  breakage  points 

■  Right  knowledge  foundation  is  critical  to  downstream  utility  & 
quality 

■  Systems  thinking  is  needed  to  ‘rise  above’  limitations  of  scope 
perspectives 

■  Need  for  speed  often  overrides  process  discipline 

■  Disaster  will  strike  if  the  foundation  is  not  properly  laid  early  in  the 
game 
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What  Impacts  Are  DoD  Seeing  Today? 


Raytheon 


■  System  complexity  has  grown  dramatically  since  the  early  cold  war 

—  Program  schedules  grew  from  3-8  years  to  greater  than  10  years 

-  Cost  growth  ranges  from  45%  to  a  staggering  100+% 

■  Of  1 1  major  programs  reviewed  by  the  GAO,  8  had  quality 
problems  attributed  to  systems  engineering  deficiencies 

■  Insufficient  systems  engineering  is  applied  early  in  the  life  cycle, 
compromising  the  foundation  for  initial  requirements  and 
architecture  development 

■  Requirements  are  not  always  well-managed,  including  the  effective 
translation  from  capabilities  statements  into  executable 
requirements  to  achieve  successful  acquisition  programs 


Sources: 

Pre-Milestone  A  and  Early-Phase  Systems  Engineering:  A  Retrospective  Review  and  Benefits  of  Future  Air  Force 
Acquisition,  2008  (ISBN:  0-309-1 1476-4) 

Increased  Focus  on  Requirements  Oversight  Needed  to  Improve  DoD’s  Acquisition  Environment  and  Weapon  System 
Quality,  February  2008  (GAO-08-294) 

NDIATask  Report:  Top  4  Systems  Engineering  Issues  within  DoD  and  Defense  Industry,  26-27  July  2006 
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Raytheon 

How  Can  Mission  Analysis  Help? 


■  Sound  understanding  of  the  mission  is  necessary  for  building  the 

right  mission  knowledge  foundation 

-  For  solving  the  right  problem  and  close  mission  capability  gaps 

-  For  creating  credible  operations  concept  and  alternative  solution  concepts, 
architectures,  and  requirements  (pre-Milestone  A  through  system  development) 

-  For  aligning  Government-Contractor  goals 

■  Insufficient  mission  analysis 

-  May  find  contractors  selling  what  they  have  in  their  inventories  instead  of  what 
is  needed  to  solve  the  problem 

-  May  cause  us  to  find  out  too  late  that  while  we  meet  stated  requirements,  we 
however  do  not  meet  mission  needs 


Mission  Needs  Are  'North  Star’  for  Systems  Engineering 
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Customer  Success  Is  Our  Mission 


Mission  is  the  Context  for  SE 


Raytheon 

Customer  Missions  and  Mission  Need  Statements 


Federal  Enterprise  Architecture  (FEA) 


DoD  EA  Reference  Models  (RM)s 


r 


DoD 


1— 

Business 

Mission  Area 

Warfighter 
Mission  Area 

Intel 

Mission  Area 

Enterprise  Information  Environmerit(EA)  Mission  Area 


Global  Information  Grid 


Think  Mission  1st 
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Mission  Need  Statements  Address  Mission  Raytheon 
Capability  Shortfalls 


■  1.  Administrative  information 

•  2.  Impact  on  Mission  Areas 

■  Briefly  describe  the  impact  of  the  capability  shortfall  or  technological 
opportunity 

■  3.  Needed  Capability 

■  Describe  the  functional  capability  needed  or  technological  opportunity. 

■  4.  Current  and  Planned  Capability 

•  5.  Capability  Shortfall 

■  6.  Impact  of  Not  Approving  the  Mission  Need 

■  7.  Benefits 

•  8 .  Timeframe 

•  9.  Criticality 

u  10.  Long  Range  Resource  Planning  Estimate 
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Customer  Success  Is  Our  Mission 


Mission  Analysis  -  Building  the  ‘Right’ 
Mission  Knowledge  Foundation 


JCIDS  Phasing  and  ‘Early’  SE 


Raytheon 


A 


(Program  Initiation) 


A 


DoDI  5000.2,  May  2003 


Concept 

Refinement 


Concept 

Decision 


/  Technology  | 

V  Development  J 

System  Development 
&  Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

Design 

/\  Readiness 
'v  Review 

.  FRP 

LRIP/  /\  Decision 

IOT&E  v'  Review 

stems  Acquisition 

Systems  Acquisition 

Sustainment 

DoD  Acquisition  Lifecycle 
Areas  of  Opportunity  to  Lay  Success-Oriented  SE  Foundation 


■  “Systems  engineering  is  the  overarching  process  that  a  program 
team  applies  to  transition  from  a  stated  capability  need  to  an 
operationally  effective  and  suitable  system”  (DoD  5000  series), 

■  Concept  Refinement  and  Technology  Development  phases  provide 
opportunities  to  work  collaboratively  with  customers  and  other  mission 
stakeholders  to  understanding  their  needs  and  their  environments 

Early  SE  is  Required  to  Effectively  Transform  Capability 
Gaps  into  an  Operationally  Valid  Mission  Solution 


Adapted  from  Raytheon  SE  Symposium  presentation  by  Adrienne  Rivera 
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Early  Systems  Engineering: 

Extending  the  Systems  Engineering  “V” 


Raytheon 


Capture  the  Customer’s  Vision 
“Understand  the  problem” 
“Define  the  need” 

“Measure  Capabilities” 
“Assess  Alternatives” 


“Build  this” 


Decompose 


Drives  us  to  answer  the  case  for  change  (i.e., 
business  case) 

What  are  we  not  doing  well  enough  today? 
What  must  we  do  better  tomorrow? 


Drives  us  to  identify  the  ‘right’  change 
What  change  makes  the  most  impact? 

Where  does  the  change  make  the  most 
impact? 

How  do  we  measure  improvement? 


Allocate 


Integrate 


Extended  ‘V’  Yields  the  Mission  Context  and  Change  Drivers 


Adapted  from  Raytheon  SE  Symposium  presentation  by  Adrienne  Rivera 
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Mission  Analysis  Implements  Early  SE 


Raytheon 


Milestones 


DoD  Lifecycle 
Phases 


Concept  Definition 


Milestone  A 


)  JCIDS  Capability  Based  Assessment  )Concept  Refinement  Phase) 


DoD 

Joint 

Functional 

Functional 

Functional 

Best 

Analysis 

Preferred 

Strategic 

Operations 

Area 

Needs 

Solution 

Materiel 

of 

System 

Guidance 

Concepts 

Analysis 

Analysis 

Analysis 

Approach(es) 

Alternatives 

Concept 

Mission 

Analysis 

Activities 


Mission 

Functional 

Concept 

Capabilities 

Solution 

Development 

Analysis 

Analysis 

•  Research  Doctrine 

•  Identify  Mission  Areas 

•  Understand  Current  Mission 
CONOPS/Capabilities 

•  Analyze  Mission  Gaps 

•  Generate  “As-Built”  Architecture 

•  Analyze  Constraints  (DOTMLPF) 

•  Create  Mission  Needs  Statement 

•  Assess  Technology 


•  Establish  Mission  Concept 
Characteristics 

•  Identify  Candidate(s)  Concept 
Capabilities 

-  Non-Materiel  and  Materiel 

•  Write  CONOPS  for  Candidate(s) 

•  Preliminary  “To-Be”  Mission 
Architecture 

•  Identify  MOEs/MOPs 

•  Prepare  Draft  ICD/DCR 


•  Identify  System  Concept  Trades 

•  Perform  Mission  Concept 
Analyses 

•  Write  Advocacy  CONOPS 

•  Develop  System  Architecture 

•  Identify  KPPs 

•  Prepare  Draft  CDD 

•  Prepare  Technology  Strategy 

•  Analyze  Cost-Benefit  &  Impacts 


•  Raytheon’s  REAP  process  guideline 
Enablers  •  Research  documentation  template 

*  Strategic  intent  template 

*  Quality  Functional  Deployment 

•  Zachman  template 

•  Mission  Concept  Document  template 

*  End-to-End  Mission  Level  Simulation 


•  Raytheon’s  REAP  process  guideline 

•  DoDAF/MoDAF  artifacts 

•  Mission  Concept  Document  template 

•  End-to-End  Mission  Level  Simulation 


*  Raytheon’s  REAP  process  guideline 

*  DoDAF/MoDAF  artifacts 

*  Man-in-the-Loop  Simulations 

*  SW/HW-in-the-Loop  Simulations 


Mission  Analysis  is  the  Foundation  Activity  of  Early  SE 


Adapted  from  Raytheon  SE  Symposium  presentation  by  Adrienne  Rivera 
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Tools  of  the  Trade 


Raytheon 

Customer  Success  Is  Our  Mission 


Raytheon  Enterprise  Architecture 
Process  (REAP)  Overview 


Raytheon 


Begins  with  understanding  the  mission 
and  mission  context 

A  systems  architecting  process  extended 
with  enterprise  architecting  support 


A  wrapper  around 
established  industry  and 
government  standards  to 
“connect  the  dots” 


Reinforced  through  strict  certification 
process 


DODAF 


FEAF 


Zachman 


TOGAF 


AT  AM® 
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REAP  Activities 


Mission 

analysis 

activities 

addressed 

here 


Raytheon 


■  Enterprise  Understanding 

-  Understand  the  Customer’s 
problem,  mission  gaps, 
constraints,  and  context 

■  Architecture  Planning 

—  Define  the  REAP-guided  work  to 
the  appropriate  level 

■  Mission  Architecting 

-  Document  the  Mission  and 
Operational  Architecture... not  the 
Technical  Architecture 

■  Technical  Architecting 

-  Define  the  Technical  Architecture 
solution  from  the  Mission 
Architecture  context 

■  Architecture  Validation 

-  Validate  the  content  and  utility  of 
the  architecture 
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Workshops,  Mission  Analysis,  and 
Mission  Experts 


Raytheon 


Formalizes  Mission  Analysis  phase 
for  large,  complex  programs 

Pilots  have  shown  that  workshops  are 
good  approach 

Ensures  strong  alignment  with 
Mission  Experts 

Uses  template  for  Data  Capture 
(AV-2s,  AV-1 ,  QFDs,  etc.) 

Captures  mission  definition,  gaps, 
challenges,  timeframe  for  target 
architecture 

Stakeholders  may  desire  to  validate 
output  and  identify  any  actions  before 
proceeding  to  downstream  activities 
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Mission  Area  Quality  Functional 
Deployment  (QFD)  Template 


Raytheon 


O  CD 

r  o 


o 

c 

ro 

r 

o 

PACOM Fires  Mission)  .i 

0) 

> 

43 

03 

CD 

01 

mission 

Capability  1 

CM 

>* 

CO 

>* 

^1" 

>* 

UO 

>* 

CO 

>* 

>> 

00 

>> 

03 

>* 

if) 

>? 

‘E 

T_ 

O 

CL 

Cl 

O 

CD 

D) 

03 

L— 

CD 

< 

Weighted  Score 

_Q 

03 

Q_ 

03 

O 
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03 

Q_ 

03 
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03 

Q_ 
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_Q 

03 

Q_ 

03 

O 

_Q 

03 

Q_ 

03 

O 

_Q 

03 

Q_ 

03 

O 

03 

03 

Q_ 

03 

O 

Scenario  1  (Use  Case)  -  as  is  state 

0 

0 

Desire  State 

Scenario  2  (Use  Case)  -  as  is  state 

0 

0 

Desire  State 

Scenario  3  (Use  Case}  -  as  is  state 

0 

0 

Desire  State 

Scenario  4  (Use  Case)  -  as  is  state 

0 

0 

Desire  State 

Scenario  5  (Use  Case)  -  as  is  state 

0 

0 

'  Desire  State 

Scenario  6  (Use  Case)  -  as  is  state 

0 

0 

Desire  State 

Scenario  7  (Use  Case)  -  as  is  state 

0 

0 

Desire  State 

Scenario  8  (Use  Case)  -  as  is  state 

0 

0 

Desire  State 

Scenario  9  (Use  Case)  -  as  is  state 

0 

0 

Desire  State 

0 

0 

1  is  low  correlation 
10  is  high  correlation 


Simple  Tool  to  Correlate  Mission  Needs  &  Capabilities 
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Applied  Mission  Area  QFD  Example 


Find,  Fix,  T rack  Individuals  of 
Interest  -  As  Is  State 

Locating  "JFC's  Most  Wanted 
People" 

Relative  importance  to  mission 

SIGINT 

IMINT 

Video 

HUMINT 

MASINT 

Multiple  Intel  Source  Aggregation 

KM/KD 

Open  Source  Exploitation 

Cyberspace  Exploitation 

Biometrics 

Resource  Management 

Average  Opportunity  score 

Weighted  Score 

Get  Tip  from  Sources  (Forces  in 
Contact,  Other  Govt  Agencies,  LE, 
SOF,  Open  Source,  Alliance 

Partners) 

6 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

30 

Desire  State 

6 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

48 

Identify  Target 
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4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

36 

Desire  State 

9 

6 

6 

6 

6 

6 

6 

6 

6 

6 

6 

6 

6 

54 

Confirm  Target  (in  Probability 

Terms) 

7 

7 

7 

7 

7 

7 

7 

4 

4 

4 

4 

4 

6 

39 

Desire  State 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

49 

Fix 

6 

4 

4 

4 

4 

4 

4 

0 

0 

of 

oA  o 

2 

13 

Desire  State 

6 

6 

6 

6 

6 

6 

6 

3 

3 

A 

6jf3 

5 

29 

Track 

7 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

56 

Desire  State 

7 

7 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

55 

Gather  Additional  Situation 

Awareness  Info  As  Needed 

4 

6 

6 

6 

[  6J 

j  6 

6 

6 

6 

6 

6 

6 

6 

24 

Desire  State 

4 

6 

6 

61 

6 

6 

6 

6 

6 

6 

6 

24 

Discern  Intent 

7 

2 

2 

2 

[2 ] 

\2 

2 

2 

2 

2 

2 

2 

14 

Desire  State 
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8 

8 

8 

81 

1  8l 

18 

8 

8 

8 

8 

8 

8 

56 

Tag 

5 
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3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

15 

Desire  State 

5 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

35 

Identifies  the  Best  ‘Focus  Area’  Opportunities 


Page  21 


Raytheon 

Mission  Analysis  Feeds  Back  to  the  Mission 


7.  Administrative  Information 

2.  Impact  on  Mission  Areas 

Briefly  describe  the  impact  of  the  capability  shortfall  or  technological 
opportunity 

3.  Needed  Capability 

Describe  the  functional  capability  needed  or  technological  opportunity 

4.  Current  and  Planned  Capability 

5.  Capability  Shortfall 


6.  Impact  of  Not  Approving  the  Mission  Need 

7.  Benefits 

8.  Timeframe 

9.  Criticality 

10.  Long  Range  Resource  Planning  Estimate 


Mission  analysis 
activities  and 
artifacts 
address  items 
2  thru  5 


Page  22 


Acronyms 


Raytheon 


1)  BRM  -  Business  Reference  Model 

2)  DOTMLPF  -  Doctrine,  Organization,  Training,  Materiel,  Leadership  &  education, 
Personnel,  Facilities 

3)  DRM  -  Data  Reference  Model 

4)  FEA  (Federal  Enterprise  Architecture) 

5)  PRM  -  Performance  Reference  Model 

6)  SRM  -  Service  Component  Reference  Model, 

7)  TRM  -  Technical  Reference  Model 

8)  UJTL  -  Unified  Joint  Task  List 
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Bio  -  John  T  McDonald 
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John  T  McDonald 

(John_T_McDonald@raytheon.com) 

>BS  in  Mathematics 
>BS  in  Computer  Science 
>MS  in  Physics 
>MS  in  Computer  Science 

Raytheon 

>RTN  Six  Sigma  Expert 
> Raytheon  Certified  Architect 
>Chief  Engineer  /Chief  Architect  IIS 
>RTN  Garland  Site  Council 
>RTN  IIS  Technology  Team 

>University  of  Texas  At  Dallas  Industry  Advisory  Board 


Summary  of  Experience 

John  has  close  to  25  years  of  experience  in  Intelligence  Community  and  DoD  Software  and  Systems 

Engineering.  John  has  served  as  lead  and  chief  engineer  on  numerous  systems  and  led  an  organization 
of  aprox  100  SW  Systems  Engineers  for  over  7  years.  John  also  lead  the  Object  Technology  Center  at 
Garland  for  5  years  in  the  early  and  mid  90s. 

John  is  currently  the  Chief  Engineer  and  Chief  Architect  of  IIS.  John  was  a  founding  member  of  the  RTN 
Architecture  Review  Board  and  formed  a  team  that  planned  and  realized  the  initial  REAP  (Raytheon 
Enterprise  Architecture  Process)  which  is  the  RTN  wide  architecture  process  and  methodology. 
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Bio  -  David  W  Rhodes 


Raytheon 


David  W  Rhodes 

(dwrhodes@raytheon.com) 

>BS  in  Computer  Science 
>MS  in  Systems  Management 
>DMSC/DAU  Advance  Program  Managers  Course 
>PMI™  Project  Management  Professional 

Raytheon 

>RTN  Six  Sigma  Expert 
>RTN  IIS  SE  Council  Co-chair 

>Colorado  State  University  Industry  Advisory  Council  (ISTeC-IAC) 


Summary  of  Experience 

David  Rhodes  has  worked  at  Raytheon  Space  Systems  in  Aurora,  CO  since  2001  and  is  currently  the  IIS 
Systems  Engineering  Council  Co-chair  and  a  member  of  the  Raytheon  corporate  Systems  Engineering  & 
Technology  Council.  David  has  over  20  additional  years  in  the  aerospace  industry  performing  in  a 
variety  of  mission  analysis,  systems  engineering,  program  management,  and  business  development 
roles.  David  is  a  graduate  of  the  DSMC  Advanced  Program  Manager’s  Course  and  Systems 
Engineering  Management  course.  David  has  an  MS  in  Systems  Management  from  the  University  of 
Southern  California  and  a  BS  in  Computer  Science  from  the  University  of  Maryland.  David  is  also  a 
member  of  the  Industry  Advisory  Council  for  Colorado  State  University’s  Information  Systems  and 
Technology  Center. 
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Innovation  Cell 

Engineering  Solutions  for  Fleet 

Readiness  Centers 


William  Birurakis 

Senior  Vice  President,  Engineering 
Pioneering  Decisive  Solutions,  Inc. 
240-298-7124 


Background 


•  The  US  Navy’s  NAE  has  in  it’s  inventory  slightly  more 
than  3,700  aircraft  (we  had  over  6500  in  1990).  There  are 
more  than  90  T/M/S  (type/model/series)  aircraft  in  the  Navy 
and  US  Marine  Corps  inventory. 

•  The  NAE  (both  Navy  &  Marine  Corps)  fly  more  than  1 .2 
Million  flight  hours  per  year  at  a  cost  averaging  a  bit  over 
$4,400  dollars  per  hour. 

•  From  a  sustainment  standpoint,  the  cost  to  provide 
everything  it  takes  to  enable  and  provide  this  level  of 
operations  and  associated  maintenance,  logistics  and 
engineering  exceeds  $  6  Billion  dollars  per  year  (not 
including  new  /replacement  aircrafts  and  associated 
systems)  and  many  thousands  of  highly  skilled  people  of 
various  skills 


Pioneering 
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Challenge 


The  Naval  Aviation  Enterprise  (NAE)  is  under  extreme 
pressure  to  achieve  ‘more  Cost-Wise-Readiness’.  This  is  a 
result  of  a  clear  understanding  that  the  strain  on  our  Navy  / 
Marine  Corps  NAE  during  current  times  is  extreme  and  that 
many  of  our  aircraft,  associated  weapons  systems,  and  the 
systems  that  support  them  are  getting  older  and  must  be 
replaced  and/or  modernized.  With  this  in  mind,  it  is 
imperative  that  the  Navy,  and  specifically  the  NAE,  seek 
innovative  ways  to  change  the  way  things  are  done  in  order 
to  achieve  more  ‘effectiveness  and  efficiency’  in  a  manner 
such  that  resource  dollars  can  be  freed  up  for 
modernization.  The  objective  has  to  be  to  achieve  exactly 
the  right  degree  of  readiness;  i.e,  not  too  much,  not  too 
little.  The  NAE  ‘is’  in  fact  doing  this. 


Pioneering 


Solutions,  Inc. 


Transformation  to  FRCs 


he  Naval  Aviation  Enterprise  (NAE)  is  transforming  the  way  it  performs  its 
Depot  and  non-deployable  Intermediate  levels  of  maintenance  by  adopting  the 
Fleet  Readiness  Center  (FRC)  concept.  In  fact,  this  initiative  was  a  part  of  the 

Base  Realignment  And  Closure  (BRAC)  process  accomplished  2005. 

• 

Per  GAO  analysis,  the  FRC  initiative,  if  fully  implemented  in  a  successful 
manner,  will  provide  the  highest  recurring  cost  saving  of  any  of  the  198  DoD 
BRAC  2005  initiatives  (ref:  GAO-rpt-159  dated  Dec2007  see  page  54). 

he  FRC  initiative,  during  it’s  first  two  years  of  implementation,  has  achieved  it 
savings  /  cost  avoidance  targets  and  these  have  been  reported  to  Navy 
leadership  as  well  the  GAO. 


hat  said,  the  FY09  target  increases  dramatically  and  the  FRC  initiative  will 

require  significant  efforts  to  actually  ‘do’  all  thal  is  required.  . 

H  &  pioneering  Decisive  Solutions,  Inc. 


Avionics  Rapid  Action  Team 


ddressing  the  thinking  and  efforts  of  the  NAE  (Naval  Aviation 
Enterprise)  to  improve  the  way  we  are  ‘providing  timely  engineering 
and  logistics  support’  to  aviation  Fleet  Readiness  Centers  that 
accomplish  the  level  II  and  level  III  aviation  maintenance  that  supports 
the  Navy’s  operating  aircraft  and  the  associated  weapon  and  support 

systems. 

• 

he  ‘Imagineering’  associated  with  the  ARAT  (Avionics  -  Rapid  Action 
Team)  is  to  deliver  to  the  FRC’s,  ‘expedited  and  focused  engineering’ 
based  upon  ‘boots  in  the  shop’  and  a  direct  and  symbiotic  relationship 
that  changes  the  way  we  identify,  then  correct  deficiencies  including 
the  alteration  of  the  associated  business  and  maintenance  processes. 
This  includes  ‘enhancing  cost  effectiveness’,  but  also  ‘system 
performance’  plus  ‘system  reliability’  or  ‘time-on- wing’. 


ey  to  this  effort  is  the  ‘measurement’  of  what  is  or  is  not  being 

accomplished  as  well  as  how  the  changes  were  made  and  can  b 
replicated  and  sustained.  pioneerinfltbeci.ive  Solutions,  Inc.  ■ 


Exploration 


ill  provide  an  explanation  of  what  has  been  achieved 
through  ARAT  at  FRC  West  located  at  Lemoore 
California  while  working  on  FA- 1 8  radar  systems 


hile  ARAT  ‘is’  focused  on  specific  achievements 
related  to  improvements  in  the  domain  of  the  FA- 18 
Hornet  radar,  the  prime  objective  is  to  prove  the 
hypothesis  that  improvements  are  possible  to  the 
methods  the  NAE  uses  to  provide  logistics, 
engineering  and  maintenance  support. 
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ARAT  Innovation  Cell  Approac 


•  Innovation  Cell  was  created  to: 

-  Identify/Solidify  Objectives 

-  Determine  Appropriate  Means  of  Measurements 

-  Generate  Approaches  to  meet  Objectives 

-  Measure  Results 

•  Many  areas  covered  for  Objectives  and  were 
boiled  down  to  two  primary  measure  of 
effectiveness: 

-  Time  On  Wing  (TOW)  &  affects  to  RFT 

-  Cost  Avoidance  /Savings 


Pioneering 
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ARAT/COE  Benefits 


POWER  SUPPLY  RADAR  RECEIVER  DATA  PROCESSOR 
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First  Pass  Yields 


]  Total  Iterations  i  i  Repair  Actions  i  i  Single  Iteration  Repair  Multi-Iteration  Repair —♦—FPY 
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FLEET 


NT  Time  On  Wing  Example 


ANT  TOW  05-07  AVG 
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ANT  Bad  Actors  example 


SERNO 

MHRS 

IMAs 

TFG857 

341.1 

11 

PUV938 

56.6 

10 

RLP327 

103.4 

8 

SVG709 

173.4 

8 

TAC756 

159.3 

8 

PDR863 

71 

7 

PUV887 

58 

7 

QGR158 

36.7 

7 

RLP349 

58.7 

7 

SQW670 

66.4 

7 

TNW153 

203.9 

7 

TNW999 

136.2 

7 

NUD579 

198.7 

6 

RLP346 

267.8 

6 

RTP388 

139.5 

6 

SAZ461 

79 

6 

QGR017 

107.6 

5 

QXC129 

76.1 

5 

REU307 

23.6 

5 

RLP320 

108.7 

5 

RLP328 

178 

5 

SAZ475 

440.9 

5 

SAZ512 

22.1 

5 

TFG838 

277 

5 

TFG868 

57.2 

5 

TNW041 

27.8 

5 

TNW050 

49.8 

5 

TNW116 

61.2 

5 

TNW229 

19.3 

5 

Only  Top  ANT  Bad  Actor  Serial  Numbers  are 
indicated  in  this  slide. 

Bad  Actors  =  13.1%  of  total  ANT  S/Ns  processed, 
33.6%  of  ANT  IMAs,  and  33.5%  of  ANT  IMA  MHRs. 
Poor  Performers  =  22.6%  of  total  ANT  S/Ns 
processed,  31.3%  of  ANT  IMAs,  and  34.4%  of  ANT 
IMA  MHRs. 

Total  =  35.7%  of  ANT  S/Ns  processed,  64.9%  of 
ANT  IMAs,  and  67.9%  of  ANT  MHRs. 


ANT  1-Level  Repair  Data  2005-QTR3  to  2007-QTR2 
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ANT  Time  Or 


ANT  TOW  Bad  Actor  Populations 


ANT  TOW  Bad  Actor  Populations 


Pioneering 


ANT  Bad  Actor  S/N  were 
removed  from  Time  on  Wing 
calculations  based  on  top  25% 
of  ANT  Bad  Actor  population 
and  at  the  50%  and  100% 
populations  from  FY2004- 
FY2007  QTR2.  FY2002-FY2003 
was  used  for  baseline 
comparison  of  trend. 


A799  Rates  Unacceptable  & 
Opportunity  for  EVHMS 


BCM  Cost  Savings 


BCM  Cost  Summary 


•  FY07  AVDLR  prices 
used  in  calculations 

•  Does  not  include  R&R 
Support  to  other  Sites 
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What 

Qty 

AVDLR 

MHRS 

FY05 

WRA 

101 

$3,570,179 

5,695 

SRA 

471 

$4,598,637 

2,525 

$8,168,816 

FY06 

WRA 

56 

$2,189,037 

1,833 

SRA 

404 

$3,936,383 

2,866 

$6,125,420 

Data  Source:  DECKPLATE 

1  FY07 

WRA 

14 

$586,879 

301 

SRA 

383 

$3,627,909 

160 

$4,214,788 

* 

Bad  Actors  Progra 


Bad  Actors 
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Tx  MQJ-618 


Number  one  Bad  Actor  transmitter  in  the  fleet 
FY05-FY07 

-  24  Failures  in  two  years 

-  Reworked  by  artisans  May  07 

Stayed  in  aircraft  11  months  before  failure 

-  >260  transmit  hours 

-  Previous  5  maint.  actions  had  a  total  of  11 
operating  hours 
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Example  RR  TNW-608 


RR  TNW  618 

23  effective  Y-Codes  (never  stayed  out) 

Reworked  stayed  out  llmonths 

$140,024.00  Cost  avoidance  by  avoiding 
continuation  of  scenario. 
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Integrated  Test  Bench  (ITB)  Benefits 

FRC  West  Lemoore 


ITB  Uses 


Y-coded  WRA's  (Repeat  offenders  for  same  fails). 

CASS  TPS  not  available  but  supported  by  ITB. 

Data/Arithmetic  problems  undetected  by  CASS 
simulation. 

Bad  Actor  processing. 

CASS  improvement  through  ITB  test  validation. 
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Integrated  Test  Bench  Statistics: 
Total  Cost  Savings  from  12/5/08. 


Total  ITB  Benefits  $651,853 
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FRC  Mid  Atlantic  Critical  El 


El  Investigation 

-  1  yr  from  fleet  intro  of  Spur  Corrections  (FST  Lead 
Time).  FRC  East  will  be  eliminating  spurs  from 
their  RRs  while  in  repair  cycle. 

—  RADAR  Receiver  Spur  Root  Cause  Analysis  (Troy 
Gordish).  Local  Oscillator  failure  mechanism. 
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TOW  Savings 


Quantity  to  Quality  based 
maintenance  and  benefits 


TOW  Benefit 


•  TOW  Cost  =  Total  Cost  of  Repair  =  $44M 

•  TOW  Increase  =  Reduction  in  Repairs 

•  Example  =  100%  Increase  in  TOW  =  50% 
Reduction  in  annual  cost  of  repair  =  $22M 
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TOW  Benefit 


•  Can  maintenance  practices  change  TOW 

•  Yes  COE  supported  systems  are  running 
approx  20%  higher  TOW  than  rest  of  Fleet 
which  yields  approximately  $2M/year  savings 

•  COE  supported  systems  are  costing  the  fleet 
less  from  BCM  interdiction  savings  and 
reduced  cost  based  on  higher  TOW 

•  Y  code  removals,  Bad  Actor  Program  etc. 
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FRC  TOW  Increase  Benefit 
about  $1M/Hr 


TOW  Increase  Benefit  Curves  Dollars/Hour 


— * — Total  Cost  — Savings 
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Changing  the  Deployed  Fleet  Co 


Can  the  COE  and  ARAT  efforts  change  the  cost 
of  Fleet  Repair 

Yes,  thru  local  El  driven  SW  changes  which 
improve  Fleet  Repair  Capabilities 
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APPROACHES  &  Actions  Taken 


Incorporated  Innovation  Cell  Findings 
Baselined  TOW  &  Cost 
TOW  Baseline 

—  TOW  completed 

—  MTBD  Lemoore  Card  Deck 

—  Bad  Actor  Determination  (By  SN) 

•  TOW 

•  A799  (CND) 
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APPROACHES  &  Actions  Taken 


•  Cost  Baseline 

-  Establish  Cost/Repair/PN  (in  work)(in  MYs) 

-  WIP  (in  work)  (MYs) 

-  FPY  (in  work)  (MYs) 

-  BCM  Interdiction  (in  work)($) 
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PPROACHES  &  Immediate  Actions  for  Effec 


-  Bad  Actor  Elimination 

•  Remove  small  percentage  for  initial  significant  reduction 

•  FRACAS  (i.e.,  perform  Root  Cause  Analysis) 

-  Change  SM&R  Codes/ICRL 

•  Example  Transmitter  Auto  BCM  for  Transmitter  Chassis  &  1A2  PSs 

-  Instill  process  for  History  Cards 

•  NAMP  Change  for  ETI  on  MAFs  (LT  Penrod) 

-  Scrap  Rate 

•  Investigate  Scrap  Rate  from  ARF 

•  Hard  Line  Manufacturing  (FRC) 

•  Micro  Min  instructions  and  training 

•  Potted  Chip  Removal  &  Card  Trace  Repair 
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PPROACHES  &  Immediate  Actions  for  Effec 


•  Training 

-  Teach  SMEs  how  to  read  CASS  digital  code 

-  Recommendation,  CW03  Daniels  approach  for  troubleshooting 
publications 

-  PMA-265  Training  Initiatives 

•  CND  Reductions 

-  Supplier  CNDs  under  investigation  (Tom  Henderson,  Kevin  Odel) 

•  A799  Reductions 

-  Feedback  to  O-level 

-  Feedback  to  SRA  Repair 

-  BOA  ECT  evaluation 
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PROACHES  &  Actions  Moderate  Ter 


•  Cooperative  FRACAS 

•  ADSR/Smart  TPS 

•  Process  Flow  Modifications 

-  Primary  Highway 

-  Rework  Lane 

-  Feedback  Loop  for  Improvement 
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The  Way  Its  Supposed  to  Be 


Pioneering 


Bottom  Line 


Results: 

-RE/RR/XMTR/ANT  TOWs(MTBDs)  have  increased  and 
FY08  levels  are  currently  being  calculated  by  FST. 
Expectations  are  in  the  range  of  2-3  hrs/Radar  =  $2-3M 
FY09  targets  another  4  hrs/radar=$8M  * 

-Cost  Reductions  in  AVDLR  from  BCMI  to  date>  $14M 

-Radar  COE  is  transitioning  from  Quantity  Driven  Repair  to 
Quality  Driven  Repair  Meeting  Demand 

-The  approach  utilized  for  Radar  Transformation  is  now 
being  utilized  for  other  commodities. 

-We  are  now  looking  into  integration  with  and 
implementation  of  EVHMS 


^calculations  based  on  2007  NAVICP  AVDLR 
Costs 
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Questions? 
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+  Introduction 

♦  Types  of  requirements 

♦  Verification  techniques 

♦  Selecting  techniques  for  requirements 

♦  Examples 

♦  Summary 
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Introduction 


♦  When  developing  systems,  it  is  a  good  idea  to  be  able  to  show 
the  customer  that  the  system  works 

»  Especially  if  you  want  to  get  paid 

♦  Involves  demonstration  of  compliance 

»  To  all  requirements,  individually  or  in  batches 

»  To  the  entire  system,  in  operational  environment  with  real-life 
operational  scenarios 

♦  Overall  system  “quality”  needs  to  fit  with  customer’s  range  of 
acceptability,  recognizing  that  trade-offs  are  usually  made 

♦  Need  to  construct  a  valid  argument  that  system  satisfies 
customer’s  requirements,  supported  with  sufficient  objective 
evidence 

»  A  requirement  is  verifiable  if  such  an  argument  can  be 
constructed 

♦  This  presentation  examines  some  techniques  for  this  proof 
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Qualities  of  requirements  (i  of  2) 


♦  IEEE  Std  830-1993*  defines  nine  qualities  for  requirements 
specifications 

»  Complete  -  All  external  behaviors  are  defined 

»  Unambiguous  -  Every  requirement  has  one  and  only  one 
interpretation 

»  Correct  -  Every  requirement  stated  is  one  that  software  shall 
meet 

»  Consistent  -  No  subset  of  requirements  conflict  with  each  other 

(  ^ 

»  Verifiable  -  A  cost-effective  finite  process  exists  to  show  that 

each  requirement  has  been  successfully  implemented 

»  Modifiable  -  SRS  structure  and  style  are  such  that  any  changes 
to  requirements  can  be  made  easily,  completely,  and 
consistently  while  retaining  structure  and  style. 


*  IEEE  Recommended  Practice  for 
Software  Requirements  Specifications 
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Qualities  of  requirements  (2  of  2) 


♦  IEEE  Std  830-1993  qualities  of  requirements  (cont’d) 

»  Traceable  -  Origin  of  each  requirement  is  clear,  and  structure 
facilitates  referencing  each  requirement  within  lower-level 
documentation 

»  Ranked  for  importance  -  Each  requirement  rated  for  criticality  to 
system,  based  on  negative  impact  should  requirement  not  be 
implemented 

»  Ranked  for  stability  -  Each  requirement  rated  for  likelihood  to 
change,  based  on  changing  expectations  or  level  of  uncertainty 
in  its  description 
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Agenda 


♦  Introduction 

^^4  Types  of  requirements 

♦  Verification  techniques 

♦  Selecting  techniques  for  requirements 

♦  Examples 

♦  Summary 
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Requirement  by  any  other  name. . . 


♦  We  use  many  different  words  to  refer  to  what  we  want  to  see 
in  a  system 


Grf0 

Threshold  sf>eC/f^a,Jce 


A* 


✓ 


\ 

JO* 


'Jt 


fa 


■i 

i 


Needs 


.# 


if  Sw 


/• 


/ 


♦  They  all  describe  some  desired  attribute  of  the  to-be-built 
system 

♦  When  developing  a  system  for  a  customer,  we  need  to  prove 
that  the  system  has  the  customer’s  desired  attributes 

♦  We  have  various  verification  techniques  to  provide  this  proof 
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Differences  among  requirements 


♦  There  are  many  different  types  of  requirements 

»  Each  type  has  different  verification  techniques  that  are  suitable 

♦  Planning  for  verification  starts  with  defining  the  requirements 

»  Important  to  define  requirements  such  that  they  can  be  verified 

»  A  key  IEEE  quality  attribute 

♦  As  requirements  mature  and  acquire  detail,  more  detail  about 
how  to  verify  them  can  be  added 

♦  Important  to  map  requirements  to  the  feasible  verification 
techniques  early 

»  And  mature  these  as  development  proceeds 

♦  Good,  complete,  and  unambiguous  requirements  inherently 
contain  the  information  necessary  for  verification 
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Types  of  requirements 
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Behavioral  requirements  (page  1  of  5) 

♦  Those  that  express  externallv-visible  actions  /  attributes  / 
behaviors  of  the  entity  (component,  subsystem,  system, 
unit,...) 

»  Defined  by  functional  requirements  /  functional  specifications 

♦  Verifiable  by  observing  externallv-visible  responses  from 
externally-applied  stimuli 

»  (Potentially)  measurable  by  testing 

♦  Seven  types 

»  Functional  »  Resource  utilization 
»  Interface  »  Trustworthiness 

»  Temporal  »  Usability 

»  Capacity 
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Behavioral  requirements  (page  2  of  5) 


♦  Functional  -  Input-output  behavior  in  terms  of  responses  to 
stimuli 


>  Simple  I/O  (stateless)  -  this  input  produces  this  output 

>  State-based  -  the  history  of  inputs  defines  the  output 

♦  Interface  -  characteristics  of  component’s  interfaces 


>  Peer-to-peer 

>  User  interface 
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Behavioral  requirements  (page  3  of  5) 


♦  Temporal  -  establishing  time  characteristics  of  behaviors 

»  Speed  -  rate  at  which  events  occur 

»  Latency  -  aka  delay  -  the  time  between  initiation  of  a  function 
and  its  completion 

»  Throughput  -  number  of  items  processed  (volume)  per  unit  time 

♦  Capacity  -  amount  of  information  that  can  be  handled 

»  System  operation  -  e.g.,  25  simultaneous  users 

»  System  data  objects  -  e.g.,  a  minimum  of  20,000  employee 
records 

♦  Resource  utilization  -  limitations  on  resources  available 

»  Defined  in  terms  of  hardware  and  other  items  that  provide 
resources  to  allow  the  system  to  operate 

»  e.g.,  memory  usage  (RAM,  disk,  flash,...),  processor  usage, 
communication  line  usage 
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Behavioral  requirements  (page  4  of  5) 


♦  Trustworthiness  (dependability)  -  degree  of  confidence  in 
product’s  delivery  of  functions 

»  Inherently  qualitative  -  cannot  be  definitively  proven  but  can  be 
inferred  based  on  evidence 

»  Types 

>  Reliability  -  probability  of  operation  without  failure  for  a  specified 
time  duration  under  specified  operational  environment  (e.g.,  0.001 
failures/hr) 

>  Availability  -  proportion  of  time  a  system  is  ready  for  use  over  a 
defined  period  of  time  (e.g.,  0.9999999  over  1  year) 

>  Safety-  features  that  protect  against  actions  that  could  lead  to 
harm  to  humans  or  property 

>  Integrity  of  operation  -  system  features  that  protects  against 
corruption  during  operation 

>  Protection  of  information  -  (confidentiality)  -  features  that  protect 
against  unauthorized  disclosure  of  information 
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Behavioral  requirements  (page  5  of  5) 


♦  Usability  -  the  ease  of  system  use  by  an  operator 

»  Two  different  flavors  based  interacting  agent  --  human  or  other 
systems 

»  When  applied  to  system-to-system  interfaces 

>  Deals  with  the  complexity  of  the  interfaces,  their  ease  of 
implementation,  and  their  efficiency  of  operation 

»  When  applied  to  human  operators 

>  Deals  with  the  complexity  of  the  interfaces  relative  to  the  how 
operators  can  operate  with  them,  the  ease  of  learning,  and  the 
efficiencies  with  which  operators  can  exploit  the  services  provided 
by  the  system. 

»  Usability  requirements  cannot  be  directly  verified 

>  Involve  inherently  subjective  behaviors  that  often  have  to  be 
observed  over  time  (e.g.,  via  a  usability  analysis) 
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Quality  of  construction  requirements 


♦  Attributes  of  the  product  itself  and  its  construction 

♦  Deals  with  how  product  can  be  handled,  not  its  operation 

♦  Inherently  qualitative  -  cannot  definitively  verify 

♦  Often  not  directly  observable  or  measurable 

»  Measures  exist  that  provide  insight  into  these  qualities, 

>  Help  to  infer  level  of  quality  based  on  quantitative  system  attributes 

»  Direct  measures  generally  do  not  exist 

♦  Examples: 

»  Portability  -  ease  with  which  component  can  be  ported  from  one 
platform  to  another 

»  Maintainability-  ease  with  which  product  can  be  fixed  when 
defects  are  discovered 

»  Extensibility  -  ease  with  which  product  can  be  enhanced  with 
new  functionality 
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Implementation  requirements  (page  1  of  2) 


♦  Restrictions  placed  on  developers  that  limit  design  space  and 
process  (aka  implementation  constraints,  design  constraints) 

»  e.g.,  use  of  specific  software  components 
»  e.g.,  imposition  of  specific  algorithms 

»  e.g.,  customer-mandated  architectures  (e.g.,  Joint  Technical  Architecture) 
»  e.g.,  imposition  of  certain  development  techniques 

♦  Two  general  types: 

»  Product  constraints  -  restrictions  on  the  product  construction 

>  Design  constraints  -  restrictions  on  design  styles  that  can  be  used 

>  Implementation  constraints  -  restrictions  on  coding  or  construction 

»  Process  constraints  -  restrictions  on  how  the  product  is  built 
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Implementation  requirements  (page  2  of  2) 


♦  An  implementation  constraint  to  a  system  may  be  a 
requirement  to  a  SW  component  within  that  system 

♦  While  these  are  required  characteristics  of  development  effort, 
they  are  not  characteristics  of  the  product’s  behavior 

»  But  will  likely  affect  behavior 

♦  Examples 

»  Use  of  specific  software  components 
»  Imposition  of  specific  algorithms 
»  Required  use  of  specific  designs  (e.g.,  open  systems) 

>  Technical  architectures 
»  Imposition  of  specific  coding  styles 
»  Required  application  of  specific  techniques  (e.g.,  RMA) 

»  Required  application  of  specific  unit  test  coverage  criteria 
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Programmatic  requirements 


♦  Terms  and  conditions  imposed  as  a  part  of  a  contract 
exclusive  of  behavioral  requirements 

♦  Address  development  aspects  of  product 

♦  Examples 

»  Costs 
»  Schedules 

»  Organizational  structures 
»  Key  people 
»  Locations 

♦  While  these  are  required  characteristics  of  development  effort, 
they  are  not  characteristics  of  the  product 

»  But  they  can  directly  affect  the  ability  to  achieve  product 
characteristics  (not  enough  time,  not  enough  budget) 
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Agenda 


♦  Introduction 

♦  Types  of  requirements 
^^4  Verification  techniques 

♦  Selecting  techniques  for  requirements 

♦  Examples 

♦  Summary 
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Requirements  and  verification 


♦  Each  and  every  requirement  needs  to  be  verified 

»  That  is,  need  to  be  able  to  construct  a  valid  argument  that  the 
requirement  has  been  satisfied  by  the  as-built  system 

»  Argument  needs  to  be  supported  with  sufficient  objective 
evidence 

♦  A  requirement  is  verifiable  if  such  an  argument  can  be 
constructed 

♦  There  are  multiple  techniques  to  construct  these  arguments 

♦  Each  type  of  requirement  may  require  the  application  of 
multiple  techniques  to  provide  a  full,  sufficient  argument 

♦  When  defined,  each  requirement  must  be  correlated  to  the 
approach(s)  to  be  used  to  verify  that  requirement 

♦  Note  that  ALL  requirements  need  to  be  verified 

»  Even  if  not  behavioral 
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Verification  techniques 


♦  We  define  five  types  of  verification  techniques 
»  Test 

»  Product  analysis 
»  Inspection 
»  Demonstration 
»  Process  analysis 

Verification  techniques 


Test  Product  Inspection  Demonstration  Process 
Analysis  Analysis 


Definitive  Analytic 
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Verifying  requirements  -  test 


♦  With  test,  we  execute  the  product,  challenge  with  stimuli,  and 
observe  behavior  (responses) 

»  Collect  the  responses 

»  Compare  responses  to  desired  responses  (oracle)  to  determine 
degree  of  adherence 

»  Desired  responses  specified  by  the  requirement  statement 

♦  Execution  environment  may  include  actual  operational 
environment  of  product 

»  May  also  include  simulations  of  other  systems  in  the  environment 


William  Bail 


22 


8th  Annual  NDIA  Systems  Conference  -  Oct2008  -  San  Diego,  California  -  Correlation  of  Types  of  Requirements  to  Verification  Methods 


Categories  of  test 


♦  Two  types  of  test  based  on  the  ability  to  determine 
conformance  to  requirements: 

»  Definitive 

>  Results  are  quantitative 

>  Can  be  compared  directly  to  the  requirements 

>  Results  can  be  stated  as  pass/fail 
»  Analytic 

>  For  requirements  that  cannot  be  definitively  verified 

-  Mathematical  and  other  forms  of  analysis  must  be  used  to 
make  an  argument  for  compliance. 

>  Test  results  from  one  or  more  tests  may  support  an  argument  for 
either  pass  or  fail,  but  do  not  provide  an  absolute  determination  of 
conformance. 

>  Such  arguments  serve  to  establish  the  levels  of  trust  that  can  be 
placed  on  the  system’s  performance 


William  Bail 


23 


8th  Annual  NDIA  Systems  Conference  -  Oct2008  -  San  Diego,  California  -  Correlation  of  Types  of  Requirements  to  Verification  Methods 


Verifying  requirements  -  product  analysis 

♦  Product  is  not  executed  (tested) 

♦  System  attributes  evaluated  analytically,  often  supported 
mathematically 

»  e.g.,  RMA  (Rate  Monotonic  Analysis) 

»  e.g.,  architecture  analysis 

♦  Results  used  to  create  arguments  of  compliance  for  those 
requirements  that  are  inherently  non-deterministic 

»  dependability 

»  to  establish  levels  of  trust 
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Verifying  requirements  -  demonstration 


♦  Product  is  manipulated  to  demonstrate  that  it  satisfies  a  quality 
of  construction  requirement 

♦  Such  requirements  express  certain  attributes  of  the  product 
but  not  how  these  attributes  are  achieved 

♦  e.g.,  portability 

»  A  portability  requirement  states  a  desire  to  be  able  to  rehost  a 
product  to  a  different  computational  environment  with  minimal 
effort  and  cost 

»  Usually  achieved  by  imposing  certain  design  constraints 
(modular  architecture,  low  coupling,  high  cohesion) 

>  Perhaps  separately  stated  as  a  design  constraint 

»  To  verify  that  the  product  is  portable,  a  demonstration  of 
rehosting  the  product  from  one  computer  to  another  may  be 
performed. 
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Verifying  requirements  -  inspection 


♦  Visual  examination  of  product,  its  documentation,  and  other 
associated  artifacts  to  verify  conformance  to  requirements 

♦  Often  used  in  conjunction  with  other  techniques  to  complete 
argument 

♦  Particularly  useful  for  verifying  adherence  to 
design/implementation  constraint  requirements 

»  e.g.,  a  software  component  may  be  inspected  to  verify  that 
makes  no  operating  calls  other  than  to  a  POSIX-standard 
interface 
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Verifying  requirements  -  process  analysis 


♦  Analysis  of  the  techniques  and  processes  used  by  developers 
to  determine  if  they  are  adhering  to  any  required  project 
standards  and  plans 

»  May  involve  examination  of  the  various  intermediate  and  final 
products  as  well  as  programmatic  artifacts  and  records. 
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Verification  approaches 
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Example  1  -  Reliability 


♦  Requirement  -  “The  system  shall  have  a  reliability  of  60  days 
MTBF” 

♦  Cannot  verify  definitively 

♦  A  test  can  suggest  failure  to  comply  but  not  compliance 

♦  Techniques  to  be  applied: 

»  Analytic  testing  -  to  observe  failure  rates 
»  Inspection  -  to  verify  built-in  fault  tolerance 
»  Analysis  -  to  examine  failure  modes  and  their  effects 

♦  Steps  for  creating  argument  of  compliance 

»  Define  appropriate  operational  scenarios,  agreed  to  by  customer 

»  Define  analysis  technique  for  predicting  reliability  based  on 
testing,  including  confidence  level 
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Example  2  -  Anti-tamper 


♦  Requirement  -  “The  system  shall  incorporate  anti-tamper 
features” 

»  Vague  -  requirement  needs  to  be  clarified 

♦  “The  system  shall  be  resistant  to  attacks  on  code  integrity” 

»  Better... 

♦  “The  system  shall  detect,  resist,  and  create  a  log  of  all 
attempts  to  change  the  code.” 

♦  Potential  techniques  to  apply 

»  Definitive  and  analytic  test  -  test  altered  code  to  verify  detection 

»  Demonstration  -  show  that  code  changes  are  detected  at  system 
load  time 

»  Inspection  -  ensure  that  code  check-sum  is  valid 
»  Process  analysis  -  verify  that  safe  processes  being  applied 
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Example  3  -  Open  modular  system 


♦  Requirement  -  “The  system  shall  be  an  open,  highly-modular 
system” 

♦  Vague  -  requirement  needs  to  be  clarified 

♦  “The  system  shall  be  designed  with  internal  modules  each  of 
which  is  no  larger  than  50  KSLOC  in  size.  The  interfaces  to 
these  modules  shall  be  documented  and  visible  outside  the 
system,  and  shall  be  easily  replaceable.” 

♦  Potential  techniques  to  apply 

»  Inspection  -  verify  that  the  modules  are  appropriately  sized,  and  that 
their  documentation  is  published 

»  Demonstration  -  show  that  each  module  can  be  replaced  by 
alternate  modules  with  same  interfaces  with  less  than  1  week  of 
effort. 
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Wrap-up 


♦  Planning  for  requirement  verification  must  start  early,  at  same 
time  as  requirements  are  defined 

♦  Requirements  must  be  written  with  the  goal  of  ensuring  that 
they  can  be  verified  effectively  and  efficiently 

♦  Verification  must  be  planned  for  all  types  of  requirements,  not 
just  behavioral 

♦  Techniques  need  to  be  selected  appropriate  to  the  type  of 
requirement 

♦  The  quality  of  the  requirement  statement  usually  drives  the 
effectiveness  of  the  verification 

»  Too  vague  results  in  loss  of  confidence 
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NDIA  -  EHM  Committee 

Enterprise  Health  Management 

Enabling  Integrated  Next  Generation  Decision  Support 

Joint  Alliance  and  Common  Reference  Model 

For  Effective  Vision  to  Transition 


23  October  2008  -  update 
Presented  by  Chris  Reisig 

Boeing 

Integrated  Diagnostics 


Approved  for  public  release;  distribution  is  unlimited 


Executive  Summary 


•  Common  Vision:  Pursuing  enterprise  transformation  driving  unprecedented 
level  of  value,  affordability,  supportability  and  availability 

*  Problem  Statement:  Enterprise  Health  Management ,  the  key  enabler  is  a 
complex  integration  challenge;  Significant  and  Common  barriers  exist  across 
stakeholders;  Inefficient  resource  utilization  across  stakeholders;  Not  leveraging 
legacy  transition  opportunities  with  emerging  programs/technologies;  Need  a 
paradigm  shift 

*  Proposed  Strategic  Approach:  Socialize  the  Common  Vision  for  Enterprise 
Transformation;  Provide  a  Focused  Systems  Engineering  Process  to  execute 
against;  Provide  Common  Reference  Model  for  barrier  identification,  solutions, 
road  mapping  and  resource  alignment 

•  Desired  outcome: 

-  Actively  drive  a  coalition  approach  towards  ‘doing  business  differently’ 

-  Provide  proactive  means  to  foster  communication 

-  Enhance  resource  alignment 

-  Accelerate  EHM/CBM  benefit  transition  to  the  Warfighter 


Approved  for  public  release;  distribution  is  unlimited 


Enterprise  Health  Management 


“The  capability  to  make  intelligent,  informed, 
appropriate  decisions  across  the  Enterprise  about 
design,  logistics,  maintenance  and  operational  actions 
based  on  Health  Management  Data  or  Information, 
available  resources,  acquisition  strategy,  and 

operational  demand.  ” 


Next  Generation  Enterprise  Health  Management  Decision 
Support  Solution  Targeting  Unprecedented  Value, 
Affordability  and  Continuous  Improvement 


Key  A ttributes  Include. . . . 

EHM  as  a  Design  Element;  Proactive  Advisory  Generation 
Based  on  Health  State;  Autonomic;  Planned  Maturation; 
Near  Real  Time  Updates;  No  False  Alarms 


Approved  for  public  release;  distribution  is  unlimited 

Common  Programs  &  Initiatives 

Shared  Vision,  Purpose  &  Barriers 


Prognostics  and  Health  Management 

The  capability  to  make  intelligent,  informed, 
appropriate  decisions  across  the  Enterprise 
about  design,  logistics,  maintenance  and 
operational  actions  based  on  HM  information, 
available  resources,  acquisition  strategy,  and 
operational  demand. 


SLIM  Mission  Statement 


n  Reneuueri  5  pint  of  Disc  over 


Integrate  WSIP,  CBM+,  RCMMISG-3,  RAM.  MFOQA,  FAVI,  and  AIP 
efforts.  Standardize  engineering  pro  cess  es/tools  associated  with 
improving  system  performance  monitoring  and  assessment  leading 
to  proactive  weapon  system  management  and  product 
improvement  throughout  the  system  lifecycle. 


CBM+  is  the  application  and  integration  of  appropriate 
processes,  technologies  and  knowledge-based  capabilities 
to  improve  the  reliability  and  maintenance  effectiveness  of 
DoD  systems  and  components.  At  its  core  CBM+  is 
maintenance  performed  on  evidence  of  need  provided  by 
Reliability-Centered  Maintenance  (RCM)  analysis  and  other 
enabling  processes  and  technologies. 
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EJiared  Processes  and  Planning  r-'WighnuLfr-: 
mu  he  if  Life -Cycle- 


Enterprise  Health  Management  is  the  Common  Denominator 
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Enterprise  Integrated  Value  Streams 


Sustaining  Engineering 


Product  life  cycle  must  be  considered  for  applicable  transition 


Operations/Fleet  Management 


Mfg/O  EM/Dep  o  t 


Maintenance 


Business 

Operations 


Health 
Data  and  Health  State 
Information 


Support  Equipment 
Training 
Chain 


Re  search/Designs/SA  T 


Transformation  Expected  Across  All  Elements 


Strong  Commonality  Across  Platforms 
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Program  IPT  Integration  Challenge 


Design 


Infrastructure  Imple  station  Business/Program 


Policy/Organization 


Technology 


Resource  Sponsors 


Software  Team 


LABs  Team 


Safety 


On  board  System 
Infrastructure 


R&M 


Mxt_pianninc)  LoR  Trainjng 

rsupt  Equip! 

T  Supply  | 


Integrity 


Logistics 
Support  System 


|  KPP  Analysfsl 

1  Logistics  Business  1 


Ground 

Station/Toolset 


Services/  PO 


Ops 


System  IPTs 


Contracts 


Design 


IS  off  board 
Infrastructure 


I  Operations  | 


|  OEM  Suppliers  I  |  Mission  IPT_|  i  ICAWS  I  I  Interoperability! 


Responsibility  is  distributed  across  All  domains; 
Need  a  integrated  systems  approach 
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Key  Drivers  for  Change 


•  System  supportability  and  affordability  goals/vision  difficult  to  meet  without 
PHM/CBM+;  Immature  cost  benefit  models 

•  Contractor  and  Government  organizational  structures  do  not  support  health 
management  as  a  new  systems  engineering  “discipline” 

•  OEMs/Suppliers/IPTs  not  fully  engaged 

•  Need  system  level  architectural  standards  that  integrate  application  of:  smart 
sensors  (e.g.,  IEEE  1451),  condition  monitoring  (e.g.,  ISO  13374)  and  functional 
and  global  data  and  information  exchange  (e.g.,  MIMOSA  OSA-CBM) 

•  PHM/CBM+  S&T  roadmaps  are  not  integrated  across  the  Services,  Agencies 
and  domain  IPTs  —  this  results  in  duplicate  core  efforts  with  minimal 
standardization,  reuse  and  transition;  Stakeholder  resources  not  aligned  to 
achieve  vision  effectively 

•  The  stovepipe  approach  results  in  the  “friction”  factor  of  disparate  capabilities 
across  the  enterprise  value  network — unsynchronized  technologies  will  create 
interoperability  problems,  waste  and  non-value  added  activity 
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A  Solution 


Based  on  a  broad  depth  of  practical  experience , 
observations  &  lessons  learned  across  various  industry 
CBM+/PH M/Autonomic  Logistic  initiatives  —  there  is  a 
need  for  a  systemic  transformation  across  the  enterprise 
—  to  address  common  barriers  and  accelerate  achieving 

the  intended  vision... 


...a  Joint  Enterprise  Health  Management  Alliance,  a 
focused  Systems  Engineering  Process  and  a  Common 

Reference  Model 
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The  Bridge 

Required^  for  Efficiency  and  Effectiveness 
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Focused  EHM  Systems  Engineering  Process 
Common  Reference  Model 


Needs ,  Barriers,  Expertise ,  Fundlrrgr^chedule,  Data, 


maps 


STAKEHOLDERS/SPONSORS 


Policy  &  Requirements 


Technology  Offices 


Programs  and  Platforms 


Research  Labs 


Warfighters 


Academia 


OEM/Integrators 


Suppliers 


Small  Business 
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Moving  Forward  Effectively 


•  Drive  a  Coalition  Alliance  -  (Best  of  the  Best) 

-  Socialize  needs,  lessons  learned,  solutions,  maturation  &  transition 
opportunities;  Cop  (Community  of  Practice) 

-  Comprised  of  Stakeholders  across  sponsors,  services,  agencies,  industry/small 
business,  academia,  and  International 

-  Drive  prioritize  needs,  resource  planning,  future  tasking,  standards,  education, 
policy  &  guidebook 

•  Provide  a  focused  Systems  Engineering  Process  and  Common  Reference  Model 

-  Enterprise  solutions 

-  Barrier  and  solution  identification 

-  Resource  Alignment  (Expertise,  funding,  data,  schedule,  transition  path) 

-  Integrated  and  dynamic  roadmapping 

•  Enhance  Transition  and  Transformation 

-  Legacy  platforms  benefit  from  early  transition  opportunities 


Enhanced  Transition  through 
a  Common  Approach,  Awareness,  and  Knowledge 
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Strategic  Objective  Summary 

Viable  Transition  with  Resource  Alignment 


ra&s  e  as# y  **  iiw  w 


Small  Business  or  Supplier 
Leveraged  Coalition  Roadmaps 


Reference 


Enterprise 


odd 


1m«  m  ipuTBd't  ntij&ii  icWf  tm ihamin^ 
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Resources: 

Organizations 

MANTECH 

4. 

r 

OSD 

International 

6.1  &  6.2 


SBIR 


STTR 


Professional  Organizations 


I  RAD 


PEO  Programs 


Research 


BAA 


Other 


DARPA 


Enhanced  Transition  through.... 

Alignment  of  Common  Needs  and  Leveraging  of  Resources 
Critical  Path  ID:  Integrated  Dynamic  Roadmaps 
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Summary  and  Action 


•  Emerging  or  Legacy  Programs  can  not  effectively  achieve  the  objective 
independently;  Efficiency  and  affordably  factors 

•  Common  fundamental  gaps  and  challenges  exist  across  all  stakeholders  and 
value  streams 

•  Need  focused  Systems  Engineering  process  and  Common  Reference  Model  to 
achieve  alignment  of  needs  and  resources 

Leadership  provide  advocacy  to  engage  and  align  key  stakeholders 

-  Execute  proposed  strategy 

-  NDIA  Tasking 

•  Mature  the  common  Reference  Model  and  Systems  Engineering  process 

-  Forum  to  build  the  Joint  Alliance 

-  Community  of  Practice  (i.e.  www.hmframework.org) 
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Common  Reference  Model  and 
Framework  Baseline 

Detail 
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Common  Reference  Model 


Business 

Architecture 

S&T  Capability 

Fn  restructure 

implementation 

This  pillar  actresses  the 

This  Pillar  references 

lOne  must  consider  all 

This  pillar  address  rhe 

This  pillar  address 

business,  pray  ram 

the  various 

the  technologies 

tools,  laFiSn 

die  uiffization, 

management  aspects  of 

architectures  that 

1  required  to  achieve  a 

certification,  testing 

autonomies. 

the  SE  approach. 

must  be  understood  or 

,  transi tinnahle  solution. 

required;  external 

human  interface 

Reference  Cost  Benefit, 

that  may  effect  your 

There  may  be 

dependencies 

requirements. 

risk  assessment. 

solution.  There  are 

dependencies, 

Implementation 

classification  and 

resources,  data 

several  on  and  off 

competing  solutions, 

architectures  and 

security:  policy 

classification. 

system,  distribution. 

existing  or  leveraging 

hosting  architectures 

changes; 

verification  neeits^ 

storage  and  utilization 

solutions 

maturation  path, 

guidance,  policies 

architectures. 

transition  path 

acquisition  strategies. 

Applicable  standards 

TRUMRL  strategy 

etc.  Organizational 

need  to  be  realized. 

Data  plans,  how  do 

structures;  policies  and 

you  use  it 

acauisitiofi  stratenies- 

Sysl 

tem  Engineering  Process  -  > 

Applied  per  Topic 

Capability  Needs/Gaps/  Barriers  Identified  Across  Domains 


EHM/CBM+  Top  Level  Vision  Domain  (population  example  only) 


Business 

Architecture 

s&r  capability 

infrastructure 

Implementation 

-Models 

-On  Vehicle 

-Sensors 

•Platform  Interface 

-  M  atufetion/T  r  a  nsi  ti  o  n 

Operational  Case 

■Off-system 

■Health  State  Methods 

•Logistics  System 

Path 

Business  Case 

-Distribution 

"Prognostics 

Interface 

■Process  integration 

Cost  Benefit 

-Software 

-Decision  Reasoning 

-Ground  Station 

■  OPS  Planning 

Trade-off 

-Utilization 

4 Enhancements  to 

Environment 

■Tool  Sets  /  Processes 

Safety/Engr  trades 

-Downlink 

Existing 

•  Test  &  Integration 

■Tech  Insertion 

-Resource  Alignment 

■Recording 

Toolset 

■Sustainment  Decision 

"Organization 

"Bata  Mgt  and  Support 

Support 

■Policies 

-Data  Compression 

•ATE  -  Smart  TPS 

-Repository  Solution 

-Data  Mgt  Plan 

-Throughput 

*  Distribution 

■Data  update/Mgt 

-Standards 

-  Standards^Rq  ts 

-Verification 

-Requirements 

■Data  Format 

-Certification 
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Multiple  Domain  Application 


Business 

Architecture 

S&T  Capability 

Infrastructure 

Implementation 

System  Engineering  Process  -  Applied  per  Topic 
Capability  Needs/Gaps/  Barriers  Identified  Across  Sponsorship 
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Proposed  Draft 
NDI A  Task  Approach 
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NDIA  Task  Summary 


The  NDIA  EHM  Committee  Task: 

•  Validate  and  Enhance  System  Engineering  Process  (Definition  and 
application) 

•  Evaluate  and  Test  Common  Reference  Model 

-  Test  viability  across  Key  domains  (Enterprise,  Platforms/Systems,  and 
Stakeholders) 

-  High  Level  EHM/CBM+  Gap/Needs  Summary 

•  Conduct  workshop  with  stakeholders 

-  Application  of  “Overarching  SE  process”  and  Reference  Model/Framework 
to  specific  domains  (populate  EHM/CBM+  Top  Level  Gaps) 

•  Provide  a  Task  Final  Report  with  Recommendations 

•  Products:  Report;  SE  process  Definitions  for  use;  SE 
Recommendations;  1st  Generation  gaps  towards  achieving 
CBM+/EHM;  High  level  gap/solution  set  and  recommendations 
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NDIATask  1  Milestones 


•  Form  Core  Task  Group  -  Jul/Aug  08 

•  Define  Draft  Tasking/Workshop  -  Aug  08 

•  Task  meeting  (Telecon/Virtual)  -  Sep  08 

•  Task  meeting  @  NDIA  HQ  -  1  Oct  08 

•  Task  meeting  (Telecon/Virtual)  -  Early  Nov  08 

•  Task  meeting  @  NDIA  HQ  -  Early  Dec  08 

•  Conduct  Workshop  -  28  -  30  Jan  09 

-  New  Orleans,  LA 
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NDIATaskl- 
Workshop  agenda 


•  NDIA  Workshop 

-  Jan  28  -30  2009 

-  2-1/2  day  event 

-  New  Orleans  Sheraton 


0800 

0830 


0900 

1030 

1030 

1200 

1300 

1430 

1500 

1630 


Welcome  and  Introductions 
NDIA  task  description 
•Executive  Summary 
•Workshop  Goals 
•Workshop  product  definition 
•Order  of  play 
•Logistics  (facility) 

•Terms  of  Reference  (What  is  EHM) 
“OSD  vision”  to  which  NDIA  will  contribute 
within  this  workshop 
Break 

Current  State  of  DoD  and  Industry 
(presentations) 

•Policy 

Lunch 

•Program  Perspective 
(emerging/legacy  platform) 

Break 

•Top  Level  Stakeholder  Visions 
(USAF/USN/USMC/NASA) 

Summary  results  of  Day  1 
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NDIATask  1  - 
Workshop  agenda  (Cont'd) 


0830  Review  of  day  1 

Refocus  on  WS  aoals  and  Products 
0900  Strawman  EHM  SE  process  description 

•Test  Case  overview 
•Q&A 

1030  Break 

1100  Breakout  Sessions  Introduction 

Discuss  goals,  test  case(s),  results  & 
formats 


1200  Lunch 

1300  Breakout  Sessions 


0800 

0830 


Day  3 

Breakout  instructions  /  goals/  output  def 
Breakout  Sessions 


in  “o  "0  m  end) 

o|  y  ®  *> 

id  Q  7?  2  co  _ | 

oo  ^  R  (Q  o-  i 
S  £  3  5‘  3 

( / )  0)  C/)^  © 

cdo  (D  r  c/> 
w  © 

0)  c 

ss 
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Overar 


1130 

1300 


1400 


Lunch 
Wrap  up 

Workshop  Key  outputs 
Action  Items 
Final  Report  outline 
Schedule  of  remaining  activities 
Conclude  Workshop 
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Strategic  Tasks  -  Not  Covered  by  NDIA  Task  1 

. .  ..but  will  be  covered  under  follow-on/separate  venue 

•  Alliance  Organization 

•  Tool  Demonstrations 

•  Integrated  Domain  Application 

•  Policy  Changes/Guidebook 

•  Defined  Standards 


Resource  Recommendations 
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VSE  CORPORATION 


VSE  Corporate  Overview 


9  Established  in  1959 
9  Rjblic  company  (NASDAQ :VSEC) 
o  Headquartered  in  Alexandria,  Viiginia 
*  ISO  9001:2000  registered 

»  Provides  woridwide  support  through  diversified 
engineering,  technical,  logisl  cs,  management;  and 
irribimat  on  technology  services  to  maintain  and 
modernize  equipment  and  systems 
o  Principal  clierrtsare  agencies  of  the  U.S 
G  ovemment  a  nd  other  government  prime 
contractors 


VSE  CORPORATION 


Prognostics?  What  is  That? 


<>  Prog  no  si  cs  is  an  engineering  discipline  focused  on 
predicting  the  future  condition  of  a  component 
and/ or  system  of  components. 
o  In  mostcases,  prognostic  approachesare  based  on 
the  analysis  of  failure  modes,  detecl  on  of  early 
signs  of  wear,  and  correlation  of  these  signs  with  an 
aging  profile  (or  model). 

9  Technical  a  roaches  to  prognostic  sc  an  be 
categorized  broadly  into  reliaoility  driven  and 
conditioned  based  approaches. 

9  The  VSE  approach  to  ftog  no  sties  Based  Health 
Assessment  incorporates  both  reliability  and 
condition  based  methodologies. 


VSEC0RP0RATI0NAn  Example  of  VSE’s  Prognostics 

Based  Health  Assessment  Systems 

®  F/A-18  Automated  Maintenance  Environment 

■  Integration  of  system  maintenance  resources  and 
configuration  data  and  into  an  integrated  system 

■  Diagnostics  •Prognostics  •  Health  Management 

■  Operator  Debrief 

■  IEIMs 

■  Life  Usage  Tracking 

■  Asset*  Configuration  Management/  Serial  Number  Tracking 

■  Interfaces  to  Supply  Chain  and  Maintenance  Management 
Systems 
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VSE  CORPORATION 


VSE  AME  Concept 


9  AME  is  first  instance  of  a  geographically  distributed 
information  system  that.. 

*  Supports  strategic  maintenance  planning  at 

■  Headquarters 

■  Each  support  level 

■  Frontline  tactical  maintenance  operations 

o  Open  system  integrating  framework 

■  Software  backplane  that  uniquely  supports  maintenance 
workflow  and  the  Application  Programming  Interfaces  (APIs) 
for  plug-and-play  software 

«  Biables  continuous  use  of  "best  of  breed"  C 015  components 


9  Generalized  APIs  that  ate  not  system-specific 


CORPORATION 


F/A-18  Sensors  &  Built-in  Test 
(BIT)  Provide  Foundation 


Each  individual  sub-system 
has  it  own  diagnostics,  BITor 
health  monitoring 
capability. 


A 


BIT  is  fully  integrated 
digitally  via  the  primary 
data  bus. 


The  BlTdata  is  recorded 
and  stored.  Data  is 
available  by  a 
removable  memory 
storage  unit. 


Mechanical,  Pneumatics, 
Hydraulic,  Engines, 

Structure,  &  Environment 
Systems  a  re  monitored  via 
analog  sensors. 

The  analog  signals  a  re 
converted  to  digital  and 
used  to  verify,  monitor, 
control  and  ensure  optimum 
system  performance. 

All  BIT,  Go/No-Go,  and  self 
test  data  is  transmitted  via 
the  data  busand  recorded 
to  the  removable  memory 
unit. 


CORPORATION 


Pilot  initiates  data 
stripping. 


Technician  performs 
repair  procedure  using  IETM 


AME  Work  Flow 


Pilot  performs 
debrief. 


Wo  rk  C  e  nte  r  Su  p  e  rviso  r 
a  ssig  n  ta  sks. 


Materiel  Control  transfers  item 
to  Supply  /  l-level 


CORPORATION 


Debrief  Logs  any  Additional  Discrepancies 


o  The  pilot  reviews  the  faults  identified  by  the  expert  system  and  adds  any  other 
discrepancies 


o  Maintenance  tasks  are  passed  to  Maintenance  Management  Database 
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CORPORATION 


Maintenance  Alerted  to  Aircraft  Caution 


o  The  aircraft  is  shown  with 
overdue  tasks 
o  Ihe  maintenance  tasks 
aie  shown 
■  Debrief  task  is  to 
repair  GPS  receiver 
(Condition  Based) 
p  Ihe  LLH  increase  has 
caused  an  engine 
turbine  to  go  'high- 
time',  requiring  an 
engine  removal 
(Reliability  Based) 


*  MainteniH. Tracker  [prodl  2usn1 /VSS  USN  System  Setup] 


File  Edit  Actions  Window  Help 
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Operator  Worksheet  [F/A-1 8C  -  1G5207] 


0,'D 
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mv 
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✓ 
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Id.  1 

Aircraft  j  Flights  Tasks  j  Plan  Steps  j  Faults  |  Usage  |  Event  History  [ 


Task 

Task  Defn 

Status 

Priority 

Deadline  Date 

<}© 

550  (#4  Roller  Bearing  High  Time  Repl) 

■ 

ACTV 

O/D 

06-Sep-1 998  1 7:4 

f 

GPS  Receiver  Repair 

r 

ACTV 

O/D 

11 -Sep-1 998  01  :C 

ii 


[±J 


r  Show  pending  tasks 
f7  Show  tasks  on  subcomponents 


Ready 
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CORPORATION 


Identify  Maintenance  Tasks 


o  System  shows  all  upcoming 
work  for  the  squadron  to 
Maintenance  Control 
*  Work  can  be  sorted  to 
focilitate  planning 
e  Maintenance  Control 
initiates  maintenance 
actions  based  on  the 
identified  tasks 


CORPORATION 


Execute  Maintenance  Tasks 


o  The  maintainer 
takes  the 
PEDD/PMA  out 
to  the  aircraft 
and  uses  the 
IEIMs  while 
executing  the 
maintenance 
tasks 


■^WebManual  -  Microsoft  Internet  Explorer 
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Description: 

A  stall  will  have  one  or  more  of  the  following  symptoms: 

1 .  A  popping  or  banging  sound. 

2.  Momentary  fireball(s)  out  the  exhaust. 

3.  Thump(s)  felt  in  the  cockpit. 

Refer  to  the  in-flight  engine  condition  monitoring  system  (IECMS) 
data  for  a  typical  stall  enclosed  in  figure  6.  If  inlet  temperature 
(T 1 )  returns  to  its  original  value  within  2  seconds  and 
compressor  discharge  pressure  (CDP)  returns  to  its  original 
value  within  0.5  second,  then  the  stall  is  a  'pop'  stall.  If 
compressor  speed  (N2)  rolls  back  to  55-70  percent  while 
exhaust  gas  temperature  (EGT)  rises  or  remains  above  550°C, 
then  the  stall  is  a  hung  stall.  Also,  during  a  hung  stall  the  engine 
will  not  respond  to  throttle  movement,  except  for  retard  to  IDLE  or 
OFF. 

Hung  stalls  can  also  trigger  other  MMP  codes: 

1 .  Inlet  Temperature  Caution 

The  stall  will  cause  reverse  airflow  out  of  engine  into  inlet.  This 
will  temporarily  raise  T1 . 

2.  Flameout  Caution 

This  will  be  triggered  by  CDP  dropping  below  20  psia  or  N2, 
dropping  below  60  percent.  If  a  flame  out  caution  occurs  and 
EGT  remains  between  550  and  800°C  no  flameout  has 
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CORPORATION 


Aircraft  Status  is  updated 


9  The  PEDD/PMA  upload 
installed  the  new 
engine  in  the  airc  raft 
logset 

9  The  Status  Board  now 
shows  the  airc  raft  as 
ready  to  fly 
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AME  Vertical  Integration 


*  Integrated  environment  linking 
maintenance  and  OEMs 

^Accurate  feedbackto  Logistics  Support 
ne:  fleet  status 

®Rapid  and  accurate  deliveriesof 
Maintenance  Plan  updates 

o  Total  Asset  Visibility  including  O-bevel 
activities 


On- Boa  id 
Diagnostics 


O- Level 


a 


I- Level 


SUPPLY 


can  'see'  the  demand  forpartsand 
other  resources  and  properly  trigger  the 
Supply  Chain 


9  Modular  components 

■  Can  rapidly  install  on  a  component 
by  component  basis 

o  Deployable 


Can  fully  operate  in  remote  locations 
with  no  operational  impact 
24/7  Global  User  Support 


OEM  PM  HQ 


DEPOT 


„ 

SUPPLY 


Gov't/Contractor 


Reports  &  Analysis 
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AME  Benefits 


®  Increased  Operational  Availability 

■  IT-related  improvements  have  increased  F/A-18  Readiness  by  8% 

■  Significantly  improved  understanding  of  current  status 

■  Improved  maintenance  efficiency  via  comprehensive  and 
accurate  diagnostics 

■  Ability  to  capture  and  use  status  information  for  maintenance  and 
supply  actions 

*  Improved  supply  chain  management  based  on  knowledge  of  in¬ 
field  demand  for  resources 

■  Pro  vides  time  ly  &  accurate  data  for  logistics  ana  lysis 

o  At  Reduced  Cost 


More  than  $1B  cost  savings  over  the  past  decade 
More  efficient  maintenance  labor  execution 
Improved  asset  utilization 
Sig  n  if  ic  a  n  t  ly  f e  w  e  rgood  orunkn  o  w  n  ite  m  s  f  lo  a  ting  thro  u  gh  sup  p  ly 


VSECORPORAT,ONAnother  Example  of  VSE’s  Prognostics 

Based  Health  Assessment  Systems 


74M  Aerostat 


74M  Aerostat 


Surveillance 
Radar  (SuR) 


Fire  Control 
Radar  (FCR) 


Communications  Payload 
""  (  MIDS,  CEC,  AFOI) 


Tether: 
6200  VAC 
"  400  Hz 


Tether: 
6200  VAC 
400  Hz 


Mobile 

Mooring 

Station 


'MEP-810B 

} 

Gensets 


Mobile 

Mooring 

Station 


|PCDS  Prime  Power* 
\  lpm  System ! 

>  *i  ? 

\  LPM 


MEP-810B 

Gensets 


LPM 


PS  -  Signal 
(cessing  Station 


LPM 


CPG  Comms 
&  Processing 
Group 


MPM 


MPM 


Command  & 
Control  Station 
CCS 


PCDS  Prime  Power  System i  ' 


Joint  land  Attack  Citiise  Missile  Defense  Elevated  Netted  Sensor  System  (J  LENS) 

Fire  Control  System  [FCS] 


Surveillance  System  [SuS] 
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Prognostics  Framework  Reasoning 


■  Expert  System  using  Model-Based 
Reasoning 

>  Uses  design-based  model  for  diagnostics/  prognostics 

>  Is  deterministic  model  using  “first  princ  iples"  of  design 

>  Reasons  by  dynamically  interpreting  the  inference  of 
data 

>  Reads  streams  of  data  from  variety  of  sources 

Interprets  sensors,  built-in  testand  operational  data;  to 
assess  system  health,  predict;  detect  and  isolate  faults 


Sustainment  Expert  System 


Reasoner 


Design- 

Based 

Model 


ystem  HealtJ 


Mission 

CapabjJit' 


Maintenance 

Needs 


Results  in  hea  tth  monitoring,  diagnostic  s  and 
prognostics 

cf  Can  be  embedded  (on-line,  real-time)  or  off- line 
Can  be  used  on  existing  ornew  systems 
Replaces  traditional  fault/logic  tree 
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CORPORATION 

Achieving  Design-Base  Comprehension 


1.  Accept  operational  data,  sensor,  BIT 
and  parametric  data  as  symptoms 

2.  Apply  reasoning  algorithms  to  predict 
&  diagnose  the  implication  of  out  of 
tolerance  symptoms  on  each  future  time 
point  defined  in  the  model 

3.  Identify  the  components  and  sub¬ 
systems  affected  by  predicted  failures  - 
sub-system  health 


Symptom  Data 


Prediction 

Horizon 


T=N 


4.  Identify  the  functions  and  missions  (3) 

affected  by  predicted  failures-  mission 
readiness 

5.  Identify  the  repairactions  needed  -  ,A) 

anticipatory  maintenance 


Subsystems 
Parts 
Faults 
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Diagnostic/Prognostic  Reasoning 


■  Build  a  System  Model  to  reflect  system  hierarchy 

■  Map  fa ult  propagation  and  test  coverage  in  a  Fault/ Symptom  matrix 

■  Correlate  actual  test  data  with  faults  ac  loss  system  hierarc  hy  (Intelligent 
Reasoned 


™  ‘  9'  Diagnostic  Connector  Internal  Test  Points  I 

and/or  Observables  Measurements  Used  and/or  Probe 
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Reasoning  Techniques 

collapse  the  field  of 


Cones  of  Evidence 
Produced  by  Pass  and 
Fail  Data 
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System  Model  Scope 


DESIGN  DATA 

•  Definition  of  Fferts,  Faults,  Failure  Modes,  Failure 
Tests,  Interconnectivity  and  TestCoverage 

SYSTEM  DATA  MANAGEMENT 

•  Input  Data  Definition  &  Characterization 

•  Prediction  Horizons 


TEST  SENSOR  DATA 

•  BITInputs  &  Mapping 

•  Sensor  Data  &  Mapping 

HEALTH  MANAGEMENT 

•  Detection  Algorithms 

•  Diagnostic  Coverage 

•  System  Stress  Factors 

•  Prediction  Algorithms 

•  Fault  Criticality 

•  Input  Data  Processing  &  Filtering 

•  Confidence  Factors 


Rates, 


JLEN3_Qrtoit 


MISSION  SUPPORT 

•  Mission  ftofile 

•  Function  Correlation  to  Mission  Phases 

•  Function  Criticality  to  Mission 

•  Immediate  Operator  Actions 


MAINTENANC  E  SUPPORT 

•  Repair  Item  Definition 

•  Combinations  of  Repair  Items 

•  Repair  Actions  (IE1M  Interface) 

•  F^arts  Ordering  Data 

•  PMC S Triggering  and  Tracking 
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Diagnostic/Prognostic  Reasoning 
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9 


Model-based  veasoner  maximizes 
the  infbimation  gained  from 
sensoisand  built-in  test 

Diagnostic  /  Prognostic  Reasoner 

■  Identifies  stress  and  wear 
factors 

*  Detects  and  interprets 
anomalies 

*  Determines  mission  capability 

*  Serial  Number  Tracking  - 
Determines  remaining  useful 
life  of  each  Hem 

w  Performs  condition- based 
prognostics 


Prognostic/Diagnostic  Model- 
Based  Reasoning  Algorithms 


Design- 
iBased  Model 
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"Dynamic"  Diagnostic  Capability 


•lest  Results  can  be  input 

...  in  any  older 

•no  pre-set  sequence 

...  thorn  any  source 

•opera  tor  observations,  test  instruments,  data  bus, 
data  file,  built-in  test,  automatic  test  equipment, 
system  pa nels&  displays,  etc. 

...  as  many  astestsource(s)  can  provide 

•not  restricted  to  one-at-a-time  to  traverse  fault  tree 
•zeroes-in  on  cause  of  fault(s) 

•Can  identify  multiple  faults 

...  Diagnostic  treesfollow  single-fault  assumption 

•Will  always  zero  in  on  fault 

...  Never  leaves  the  technician  hanging 

•Only  requests  tests  of  diagnostic  significance 

...  Based  upon  snapshot  of  current  fault  possibilities 


Embedded  System  Interrogation 
System  Status 
Fault  Description 
Fault  Evidence 
Maintenance  Procedures 
Troubleshooting  Guidanc 
Repair  Options 
Data  Log 
Parts  Ordering 
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VSECORPORATIONPrognostics  Based  Health  Assessment 

Functionality 


e  Integrates  diagnostic/ prognostic  results  into  a  Health  Management  System 


Q 


Q 


Makes  maximum  use  of  existing 
Sensory  BlTdata 

Provides  ftognostics 
Ana  lysis/  Reasoning 

9  Degradation  of 

sig  na  Is/ me  a  sure  me  nts  o  ve  r  time 
9  Depletion  of  consumable  items 
9  Accumulates wearfactors 
9  Engineering  correlations 
9  Tra  c  ks  p  re  ve  ntive  ma  inte  na  nc  e 
based  on  time/wear/use 
9  Serial  numbertrac king 
9  Remaining  Useful  Ufe 


o  Allows  for  integration  of  3rd  party 
prediction  tec  hniques 


o  C  ompiles,  interprets  a  nd  displays  trend  data 
o  C  reates  multiple  log  files 

o  Links  to  maintenance  systems  (IE1M,  PMCS>  Supply)  based  on  specific  fault 


CORPORATION 


3  Views  into  Health  Data 


Operator-  Am  I  OK? 
If  not  why  not? 

What  do  I  do? 


Mission  Comma  nder- 
Wfill  this  system  make  it 
through  mission  without 
failure?  Which  of  my 
systems  will  make  it 
through  the  mission? 


Maintainer-  What  repairs 
need  to  be  made?  What 
spares  do  I  need  to  make  the 
repairs?  What  are  the  repair 
procedures?  What  PMC  S  is i 
currently  due? 


/  Health  Monitoring  Version  6.0,0  Oemo 

Monitored  Systems  N. 

To  Show  Detail  of  Tree  Item:  \ 

High  light  and  Right  Click  \ 

\  CPG 

7" 

ei  ®  JLENS 
a  ■  FCR 
f  ■  Platform 
ei  F  CPG 

±j  ¥  Communications  Systems 
b  W  Communications  and  Control  Station  (CCS 
+  ■  CCS  Power  Subsystem 
-  V  CCS  Environmental  Control  System 

CCS  Environmental  Control  System. C 

►  CCS  Environmental  Control  System. C 

►  CCS  Environmental  Control  System. C 

►  CCS  Environmental  Control  System. C 
CCS  Environmental  Control  System. C 

►  CCS  Environmental  Control  System. C 
s  ■  Network  Subsystem 
±i-  ■  Processors  Subsystem 


A  I- /“M  Ij'-'I- /“M 


System  Health 


Mission  Readiness 


Maintenance 


&  Show  Sub-System  Health 

IS  how  All 

Show  Functional  Capabilities 

■"  Show  Failures  Only 

Part 

Condition 

Critical 

Communications  Systems 

Operational 

No 

Communications  and  Control  Station  (CCS) 

Impending  Failure  (1 2h  ) 

Yes 

Data  Processing  Station  (DPS) 

Operational 

No 

CORPORATION 


Operator  View  -  Real-Time  Status 
Monitoring  &  Health  Assessment 


®  Drill  down  the  hieiaic  hical  model  to  getthe  level  of  detail  desired. 


VSE 
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Mission  Commander  View 


Not  just  sensor  data 


Network  Nemr  tor  life  Nude 


Addhund  Inturmetiuri  tor  U*>  Nude 


(DEVIL  FISHO 


(ihtp:  hmlnomr  ) 


Hunn) 

■mperature  lemperatu 

»:  |  I 

A  ; 

:|  :J 

(Ta.22  to.oo 


■ij  Multiple  System  Monitoring 


mi  Monitored  Systems 

To  Show  Detail  of  Item:  Highlight  and  Fight  Click 


I 
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Impact  of  functional 
degradation  on 
specific  missions 
over  time 


-  S&  USS  George  Washington  (CVf1  *  | 

[j . |  F/A-1 8  BUNO  163456 

g . ■  F/A-1 8  BUNO  164359 

i . |  F/A-1 8  BUNO  163058 

+  |  F/A-1 8  BUNO  163432 

|i| . ■  F/A-1 8  BUNO  163434 

B . ©  USS  Monterey  (CG61) 

+  |  FI GM84D  Harpoon  Anti-S 
^  R  G  M  84D  H  arpoon  Anti-S 

+i . H  MK88  Fire  Control  Radar 

+  ■  MK1 5  PHALANX  Close  I  r 
15  5'754  Caliber  Nav< 

11  Vertical  Lauch  Sys 
SQS-53D  Hull  Mount 
jellors  j 

V  Standard  Stockless 


Jed  Water  System 
Main 

Air  System 
owave  T  ube 
nandy  [CG801 


lick  to 


....but  also  mission 
readiness  based  on  status 
of  inter-related  systems 

1 3 


USS  Monterey  (CG61 ) 


System  Health  [ 

Mission  Readiness 

|  Maintenance  | 

Mission  Detail 


Persian  Gulf  Cruise 


m 


Select  Missior  Description:  Cruise  Persian  G ulf  for  4  months.  Patrol  for  illegal  I raqi  oil  tankers.  E scort  Kuwaiti  0 il  T ankers 
through  Gulf  or  Hostile  Seas. 


|  Persian  Gulf  Crui; 


Essenti 


Mission 

Duration: 


5  Months  3  Weeks  1  Day  22  Hours 


[Mission  Start] 


I  Mission  End  I 


Combat 


2  Months  4  Months 


Electrical  Distribu 

Electrical  Genera 
Electronics  Coolir 

Fuel  Storage  and 

Hotel  Services 

Lube  Oil  Storage 

Sea  Water  Coolir 

Self  Defense 


Combat 

Damage  Control  Monitoring 
Electrical  Distribution 
Electrical  Generation 
Electronics  Cooling 
Fuel  Storage  and  Distribution 
Hotel  Services 

Lube  Oil  Storage  and  Distribution 
Sea  Water  Cooling 
Self  Defense 
Ship  Propulsion 

To  show  duration,  starting  and  ending  times  for  lime  period, 
highlight  the  junction’s  bar  area. 


□  c 


View  Errors 


Done 


CORPORATION 


Maintainer  View 
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System  Capability  (SYSCAP) 


^Health  Monitoring  Version  6.0.0  Demo 


Monitored  Systems 

To  Show  Detail  of  Tree  Item: 
Highlight  and  Right  Click 


CPG.CCS  Environmental  Control  System 

CCS  ECU  1  (Filter  Clogged) 


E 


PFR  4.0.0  02/26/07 


®  JLENS 
b  ■  FCR 

+■  ■  Heat  Exchanger  Unit 
+  ■  Rack  Equipment  Group 
+  ■  Antenna  Assembly 
B  ■  Platform 
+  ■  Aerostat 

+  ■  Mobile  Mooring  Station 
a  ■  Tether 

t  ■  Ground  Support  Equipment 
b  *  CPG 

+  ■  Communications  Systems 
-  V*  Communications  and  Control  Station  (CCS 
ra  ■  CCS  Power  Subsystem 
-  V  CCS  Environmental  Control  System 

CCS  Environmental  Control  System. C 

►  CCS  Environmental  Control  System. C 

►  CCS  Environmental  Control  System. C 

►  CCS  Environmental  Control  System. C 

►  CCS  Environmental  Control  System.C 

►  CCS  Environmental  Control  System.C 
■+  ■  Network  Subsystem 

+  ■  Processors  Subsystem 
+  ■  AFOI/GFOI  Interface 
+  ■  Optical  Surveillance 
si  ■  Data  Processing  Station  (DPS) 

*  ■  Signal  Processing  Station  (SPS) 


Additional  Displays 

Preventative 
Maintenance 


aerial  Numbers 


Fault 

Description:  Tt 
cl 


Graph  Display 


011® 


Action: 


Ci 


Failure  Time  Fra 


Failure  Predictio 


Failure  Symptorr 


To  Show  Symptot 
Highlight  item(s) 


View  Selection 


Select  one  or  more  phots  and 
click  "Refresh  Graph  " 

17  - 

ECU_Filter  (45) 


Refresh  Graph 


Show  Point  Values 


I.  Click  Unit  Color  below 
chart  for  list  of  Plot  Values 


2.  Click  Plot  Point  for  value 


r 


Done 


Done 


•SYSCAP  displays 
system  status  based 
on  the  hierarchical 
breakout  of  the 
system: 

•  System 

•  Prime  Item 

•  Critical  Item 

•  LRU 

•  Fault 

•  Failure  Mode 

•At  each  level, 
appropriate  information  is 
displayed. 

•Operator  can  drill-down 
to  investigate  health 


HMS  displays  and  interaction  were  demonstrated  at  Early  User  Assessment  in  March  2008 
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Additional  Features 


»  Preveritve  Maintenance 

®  Maintenance  and  repair  procedures  linked  to  fault 
enunc  iation 

*  Model  can  launch  IE1M  to  specific  repair  procedure  for  fault 
a  Serial  Number lia eking 

o  Interlace  to  Parts Oidering 

a  Data  bogging 

a  Validate  Sensor  Data 
f  Missing  orinvalid  data 
■  Valid  sensor  ranges 
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CORPORATION 


o  Run-lime  Software  designed 
for  embedded  applications 

9  C  Code  that  can  be  cross- 
compiled  to  any  platform 

9  Implementation  Strategy: 

■  Centralized 

■  Distributed 

■  Hierarchical 

9  Software  functions  serve  as 
building  blocks 

®  Integrate  building  blocks  to 
build  desired  functionality 

■  Design  User  Interlace  as 
desired  or  use  existing 

»  Well-documented  API 


Software  Architecture  of  the 
Health  Management  System 


Health  Management  System 
User  Interface 


Generic 


API 


Model-Based  Reasoner 


Design-Based 
Model 
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CORPORATION 


Layers  of  CBM  + 


System 

Design 


Diagnostic 

Reasoning 


Ptognostic 


Health 


Condition  Based 


Reasoning  /  Management  Maintenance 


Design-Based  Comprehension  of  System  Condition 
is  the  Most  Fundamental  Enabler  of  CBM+ 
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VSECORPORATION  Other  VSE 

Diagnostics/Prognostics  Based  Programs 


®  Navy  SPS-48E  Radar 
®  C-130  G  unship 
®  A- 10/  KC  - 135  Turbine  Engine 
Monitoring  System 
«  Kiowa  V\farrior  Mast 
Mounted  Sight 

«  Sea  wolf  Ship  Control  System 
®  Avitronics  Radar  Warning 
Receiver 


®  NASA  Remote  Power 
Controller 

9  F-16  Universal  Data 
Acquisition  System 
9  Navy  lotal  Ship  Monitoring 
(ISM)  Program 
9  Navy  Battle  G  roup 

Automated  Maintenance 
Environment  ftogia  m 
9  FAA  Wide  Area 

Augmentation  System 


CORPORATION 


VSE  Capabilities: 
T otal  Implementation  Support 

®  la ilo table  to  any  platfbmn  or  system 
o  VSE  has  the  capability  and  experience  to  bring  all 
of  the  tesoutces  together  to  foige  a  PRACTICAL, 
EXPEDIENT and,  COSTBTECUVE  solut  on: 

■  Requirements  Analysis/ Implementation  Strategy 
«  Integration  &  Middleware 

■  Legacy  Data  Capture 

if  Development  of  System  Diagnostic/ ftog nosdc  Models 
«  Installation  &  Relding 
if  liaining 

if  Fleet  Support  Team 


% 

f 

| 

3 
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Points  of  Contact 


J  erry  J  ohnson 
Marketing  Manager 
imiohnson(a)vsec  orp.com 

(757)  635-8385 

Ron  Newman 

Director,  Diagnostic  sand  Prognostics  Products  and  Services 

rdnewman(g)vsec  orp.com 

(757)  523-7291 

Terry  Chandler 

Vice  President,  Division  Manager 
td  c  ha  ndler@vsec  orp.com 

(301)  866-5139 


SSE 

Systems  and  Software  Engineering 

Enhanced  Systems  Engineering  - 
Starting  Programs  Right 

NDIA  11th  Annual  Systems  Engineering  Conference 

October  23,  2008 


Ceasar  D.  Sharper 

Systems  and  Software  Engineering/Enterprise  Development 
Office  of  the  Deputy  Under  Secretary  of  Defense 
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Enhanced  Systems  Engineering  (SE) 

•  SE  Context:  Background  /  Framework 

•  Early  SE:  . .  right  activities  at  the  right  time  . . 

-  Materiel  Solution  Analysis  Phase 

-  Technology  Development  Phase 

•  Emphasis  on  SE  the  right  time  in  the  right  way” 

-  Competitive  Prototyping 

-  SE  Design  Consideration  -  Reliability, 
Availability,  and  Maintainability 

-  Preliminary  Design  Review  (PDR) 


“Implementing  the  right  activities  at  the  right  time  in  the  right  way ” 
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Background 

Program  Roles  &  Activities 


Systems  and  Software  Engineering 


Project  Manager 

Systems  Engineer 

Stakeholder  Management 

Primary 

Support 

Planning 

Primary 

Support 

Cost  Management 

Primary 

Support 

Schedule  Management 

Primary 

Support 

Configuration  Management 

Primary 

Support 

Contract  Management 

Primary 

Support 

Concept  Selection 

Shared 

Shared 

Architecture  Development 

Support 

Primary 

Requirements  Baseline 

Support 

Primary 

Technical  Risk  Management 

Support 

Primary 

Interface  Control 

Support 

Primary 

Integration 

Support 

Primary 

Verification 

Support 

Primary 

Validation 

Shared 

Shared 
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“Proposed  DoDI  5000.02  Changes” 

Framework  for  Enhanced  SE  — 


Technology  Opportunities  &  Resources 


MS  B 


•  Entrance  criteria  met  before 
entering  phase 

•  Evolutionary  Acquisition  or 
Single  Step  to  Full  Capability 


Full  Rate 
Prod  DR 


MS  C 


Strategic 

Joint 

Guidance 

Concepts 

FAA  FNA 
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Solution 
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Incremental  Development 
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OSD/JCS  COCOM 
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Acquisition  Phases 


Technical 

Processes 


Material 

Solution 

Analysis 


MS  A  Technology 
Development 


Engineering 

Manufacturinc 


r«o  d  Manufacturing 
ivio  d  Development 

▲  and 

Demonstration 


Production 

and 

Deployment 


MS  C 

Operations 
and  Support 


Stakeholder  Requirements 
Definition 

Requirements  Analysis 


Architecture  Design 

Implementation 
Verification 
Transition 
Validation 


“Right 
activities 
at  the 
right 
time” 


Technical  Reviews 


Technical  Baseline 

Early  Systems 
Engineering 


Preferred  System  System 

System  Spec  Functional 

Concept  Baseline 


Allocated 

Baseline 


A 

CDR 

A 

TRR 

A 

SVR/ 

FCA/ 

PRR 

A 

OTRR 

A 

PCA  1 

Initial 

Product 

Baseline 

Product 

Baseline 

ITR  -  Initial  Technical  Review 
ASR-  Alternative  Systems  Review 
SRR  -  System  Requirements  Review 
SFR  -  System  Functional  Review 
PDR  -  Preliminary  Design  Review 
CDR  -  Critical  Design  Review 


SVR  -  System  Verification  Review 

TRR-  Test  Readiness  Review 

FCA-  Functional  Configuration  Audit 

PRR  -  Product  Readiness  Review 

OTRR  -  Operational  Test  Readiness  Review 

PCA-  Physical  Configuration  Audit 

ISR  -  In-Service  Review 


Materiel  Solution  Analysis  Phase 


Systems  and  Software  Engineering 


MDD 


MSA 
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SE  Activities  During  MSA  Phase  < 99E) 

SyjfefiJj  Software  Fngmeerwg 


Systems  Engineering  Processes/Documents/Plans 

■  Key  Technical  Processes 

■  Systems  Engineering  Plan  (SEP) 

■  Test  and  Evaluation  Strategy  (TES) 

■  Analysis  of  Alternatives  (AOA) 

■  Input  to  the  Technology  Development  Strategy 

■  Input  to  the  Cost  Estimate 

Assessments 

■  Program  Support  Review  (PSR) 

Technical  Reviews 

■  Initial  Technical  Review  (ITR) 

■  Alternative  System  Review  (ASR) 


SE  COP  (https://acc.dau.mil/TechRevChklst). 
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Technology  Development  Phase 


Systems  and  Software  Engineering 


MSA 


Technology  Development 


User  assessment  of  capability  needs 


Execute  Tech  • 
Maturation  ^ 


Initial 

TRA 


Requirements  to 
Preliminary  Design** 


\  f 

Develop  Feasible 
System  Design  (FD)  FD 


Draft  System 
Level  Spec  Sys  Spec 


PDR 
PDR(s)  Report 


CTE 

Risk  Reduction 

_A_  CTE* 

. . ^  Prototyping 


m _ 

tummu 


Other 

Program 

Activities 


Actions 
Depending 
on  SE  input 


Typically  executed  by  PMO  SE  Staff 

Typically  executed  by  Industry 

^  Delivered  Product 

Mandated  Preliminary  Design  Review 
Technical  Review 
☆  Technical  Assessment 

Grey  Areas  depending  on  PMO  SE  input 

- ►  Leads  to 

- ►  Informs 

- ►  Key  SE  input 


Final 

System  Spec 

SEP 

ISP 


Draft  RFP 
for  Initial 
Sys  Dev 

Initial 

AS 

TEMP 

CARD/ICE 

CCE 

APB 


TEMP 

CARD/ICE 

CCE 

APB 


Final 

♦as 
♦temp 
♦card/ice 
"^icCE 

*  ^APB 


Prototyping  for  CTE  and  for  design  may  be  independent  efforts 
May  vary  with  contracting  strategy  (e.g.,  multiple  designs) 
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SE  Activities  During  TD  Phase  09£3> 

Systems  and  Software  Engineering 

Systems  Engineering  Processes/Documents/Plans 

■  Key  Technical  Processes 

■  Competitive  Prototyping 

■  Technology  Maturation 

■  Test  and  Evaluation  Master  Plan  (TEMP) 

■  Cost  Analysis  Requirements  Description  (CARD) 

■  Input  to  the  Acquisition  Program  Baseline  (APB) 
Assessments 

■  Technology  Readiness  Assessment  (TRA) 

■  Program  Support  Review  (PSR) 

Technical  Reviews 

■  Systems  Requirements  Review  (SRR) 

■  Systems  Functional  Review  (SFR) 

■  Preliminary  Design  Review  (PDR) 


Prototyping  and  Competition 
in  the  right  way” 


Systems  and  Software  Engineering 


“Evolutionary  acquisition  requires  . . . 
Technology  development  preceding 
initiation  of  an  increment  shall  continue 
until  the  required  level  of  maturity  is 
achieved,  prototypes  of  the  system  or 
key  system  elements  are  produced,  and 
a  preliminary  design  is  completed.  . . 

“The  TDS  and  associated  funding  shall 
provide  for  two  or  more  competing 
teams  producing  prototypes  of  the 
system  and/or  key  system  elements 
prior  to,  or  through,  Milestone  B.  The 
prototypes  shall  be  representative 
platforms  reflecting  the  maturity  of 
technologies  and  integrated  system 
performance  consistent  with  expected 
capability.” 


THE  under  secretary  of  defense 

PCNTAG°N 

WASHINGTON,  DC  20301.3010 


i  *  sep  im 
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programs  provide  a  method  to  e^reise  a"id  remta 2LJ CT*,nfer‘n*  *k«le  Third,  the 
e  government  and  our  industrial  base  Founh  ■>,  ^  Cnll<-:a  <Jore  engineering  skills 
generation  of  young  scicntisis  and  cneuierr*  in  ^  peototvpe  efforts  can  attract  a  new 
of  our  Nation’s  Warfighter  Finally  iw"*  ’  w"  leCh',ic!''  H'''™’1  10  *r  needs 

o 
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Competitive  Prototyping 
“Done  in  the  Right  Way” 


Systems  and  Software  Engineering 


■  Need  to  know  earlier  on  what  will  make  the  program 
successful  and  prototype  that  (i.e.  challenges) 

■  Decide  what  is  important  -  cost,  integration,  technology, 
etc  -  and  determine  how  to  measure  /  assess  success 

■  Cost  in  prototyping  should  be  a  factor  but  the  not 
dominant  decision  point 

■  Get  domain  experts  to  assist  in  determining  what  needs 
to  be  prototyped 

■  Do  proof  of  concept  but  also  to  fill  in  the  other  holes 

■  Achieved  at  any  level  -  system  or  key  system  elements 
(sub-system,  assembly,  or  component) 

■  Prototype  the  critical  path  items  first 

■  Need  to  spend  money  smartly  up  front  -  get  smart  at  low 
burn  rate 
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Enhanced  Systems  Engineering 
SE  Design  Consideration  -  RAM 


Systems  and  Software  Engineering 


Reliability,  Availability,  &  Maintainability  (RAM) 

■  Defense  Science  Board  Report  on  DT&E  (dtd  May  08) 
recommended  to  improve  RAM 

■  DoD  Working  Group  formed  to  implement  recommendations 

■  Reliability,  Availability,  and  Maintainability  Policy  (dtd  21  Jul 
08);  Directs  Components  to  set  policy  actions  to  ensure: 

■  Collaboration  in  the  establishment  of  RAM  requirements 

■  Development  contracts  and  acquisition  plans  evaluate 
RAM  during  system  design 

■  Maturation  of  RAM  throughout  the  acquisition  life  cycle 

■  Use  of  contract  incentives  to  achieve  RAM  goals 


(http://www.acq.osd.mil/sse/dte/docs/USD-ATLMemeo-RAM-Policy-21Jul08.pdf) 
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Implementing  RAM-C 
“■■■  right  activities  at  the  right  time  ...” 


Systems  and  Software  Engineering 


Executive  summary  of  RAM-C  Rationale 
Report  submitted  with  CDD  and  CPD 


Materiel 
Development 
Decision 


Material  Solution 
Analysis 


A 


_ PDR 

Technology^^ 

Development/Prototypes 


Enqineerimj  &  Manufacturing 
Development  and 
Demonstration 


AoA 


Scopes  the 
system  solution 

or, 

EoA 


RAM-C 

Rationale 


Update  AoA 


Note:  MSB 
is  post  PDR 


reduction  and 
Deployment 


A 


l  RIP 


Materiel  Solution  analysis^ 

(AT&L/Ser vices  role) 


i  Update  AoA 

Update 

RAM-C 

Rational* 

RAM-C  Rationale  Report  Published  in  Conjunction 
with  the  Analysis  of  Alternatives  (AoA) 
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Systems  and  Software  Engineering 


Implementing  RAM-C 
"...  in  the  right  way” 


SE  Design  consideration  in  the  right  way” 

■  Template  for  Reliability  Contract  Language 

■  Sections  C,  L,  and  M 

■  Guidance  on  Performance  Incentives  for  Reliability 

■  GEIA-STD-0009,  Reliability  Program  Standard  for 
Systems  Design,  Development,  and  Manufacturing 

■  RAM  Planning  Template  by  each  Technical  Review 

■  Evaluation  Criteria  (Reliability  Program  Detailed 
Scorecard)  to  assess  a  program 

■  Early  T&E  Involvement  in  RFP  Development 

■  DoD  Reliability,  Availability,  Maintainability  and  Cost 
Rationale  Report  Manual,  October  XX,  2008 

(http://www.acq.osd.mil/sse/dte/spec-studies.html) 

“Having  performance  is  important, 
but  not  as  important  in  most  cases,  as  having  reliability” 


14 


Life  Cycle  Logistics  Flow  (RAM) 


Evaluate  Technology’s 
Ability  to  Implement 
Supportability 
Objectives  & 
Sustainment  Strategy 


Set  Sustainment 
(RAM)  Metrics  & 
Requirements 


Refine  Sustainment 
Objectives  and 
Requirements 


Define  Initial 
Supportability 
Objectives  & 
Sustainment 
Strategy 


Demonstrate 
Feasibility  & 
Mature 
Technology 


- CONOPS 

-  Analysis  of  Alternatives 
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-  Production/Quality  Metrics 
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Implement 
Product  Support 
Strategy  and 
Package 
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Features  &  Establish  Product 
Support  Package 
Requirements 
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Solution 
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☆ 
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☆ 
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▲  ▲  ▲ 

„ 

▲  ▲  A 

▲ 

User 
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Reviews 
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ASR 


SRR 


IBR  SFR 
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TRR  FRR  SVR/FCA/ 


OTRR  PCA 


ISR 


Technical 

Baseline 


Preferred 

System 

Concept 


System  System  Allocated  Product 

Specification  Functional  Baseline  Baseline 

Baseline 


Product 

Baseline 


Contract  Action 


RAM  included  in  Systems  Engineering  Tech  Reviews 


Enhanced  Role  of  PDR 


Systems  and  Software  Engineering 


MSA 

A 


SRR 


Materiel  Solution 
Analysis 


Technology 

Development 


Interpret  User  Needs, 
Refine  System 
Performance  Specs  & 
Environmental  Constraints 


grades 


Develop  System 
Functional  Specs  & 
Verification  Plan 
To  Evolve  System 
Functional  Baseline 


Functional  Baseline 


\ 


Revised  Program  Information 

•  Technical  Plans 

•  IMS/IMP 

•  Risk  Assessment 

•  Detailed  Requirements 


MS  C 

A 


FRP  DR 


<j^CDR^>  EMDD 


CPD 


pdO 


o&s 


Acquisition  Program  Baseline 


Acquisition  Strategy 


SEP  /  Technical  Planning 


Evolve  Functional 
Performance  Specs  into 
Cl  Functional  (Design  to) 
Specs  and  Cl 
Verification  Plan 


•  Sys  Allocated  Baseline 

•  Live  Fire  T&E  Waiver  request  (if 
appropriate) 

•  Risk  Assessment 
•Sys  Threat  Assessment 

•  Affordability  Assessment 

•  Cost  &  Manpower  Est. 


TEMP  /  Test  Plans 


Information  collected  from  PDR 
enhances  plans  &  estimates  to  enable' 
knowledge-based  MS-B  decision  and 
certifications 


Allocated  Baseline 


•  Cost  Estimate  (ICE//EA  -  CARD) 


•  PPP/CPI 


Information  Support  Plan 


CDD  /  Requirements 


Certifications  (e.g.,  CCA) 


PESHE 


Post  PDR  Report 


TRA  (TRL  Level  6  /  Off-Ramp) 
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■  Certification  and  Accreditation  activities  scoped  and 
identified 


■  Configuration  Management  Plan  and  procedures  scoped 
and  implemented 

■  Integrated  Master  Schedule  showing  Critical  Path 
through  Critical  Design  Review 

■  Software  Development  Plan  scoped  and  documented  at 
the  Configuration  Item  level 

■  FMECA  scheduled  to  support  System  Hazard  Analysis 

■  Modeling  and  Simulation  role  in  testing  and  life  cycle 
planning  scoped 

■  Representative  mission  profiles  finalized 


17 


(Continued) 


■  Engineering  data  requirements  needed  from  testing 
identified 

■  Data  element  identification  procedures  established  -  IDE 
procedures  established 

■  Test  Verification  Matrix  covering  subsystem  allocations 

■  Physical  properties  (i.e.,  weight,  power,  cooling,  etc.) 
allocated  to  subsystems 

■  Human  Systems  Integration  design  standards  flowed  to 
subsystems 

■  R&M  diagnostics  addressed  in  design  allocations 

■  Interface  Control  Documents  between  subsystems 
completed 
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PDR  Report  to  MDA  (35& 

System  and  Software  Engineering 


DRAFT  PDR  Report  Guidance  to  require  the  following: 

■  A  comprehensive  list  of  the  systems  engineering  products 
that  make  up  the  Allocated  Baseline,  per  the  PDR  review, 

■  A  list  of  the  participants  in  the  review,  including  the 
independent  (of  the  program)  chair,  applicable  technical 
authorities,  independent  subject  matter  experts,  membership 
of  the  Technical  Review  Board,  and  other  key  stakeholders, 

■  A  summary  of  the  Action  Items  and  their  closure  status/plan 

■  A  resulting  risk  assessment  using  a  PDR  risk  assessment 
checklist  and  readiness  to  commit  to  full  detail  design, 

■  A  recommendation  from  the  PDR  as  to  the  approval  of  the 
program’s  system  Allocated  Baseline  to  support  detail 
design. 

Proposed  Source:  DAG  para  4. 3.2.4. 2. 3 
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Enhanced  SE  Provides  key 
information  for  the  MS  B  Decision 


Systems  anti  Software  Engineering 


Enhanced  SE  contributes  to  key  MS  B  prerequisites 

■  Acquisition  Strategy  (including  core  logistics 
analysis/source  of  repair;  cooperative  opportunity;  etc.) 

■  Independent  Cost  Estimate 

■  Cost  Analysis  Requirements  Description  (CARD) 

■  Manpower  estimate 

■  Acquisition  Program  Baseline 

■  Analysis  of  Alternatives 

■  System  Threat  Assessment 

■  Technology  Readiness  Assessment  (TRA) 

■  Affordability  Assessment 

■  Selected  Acquisition  Report  (SAR) 

■  SEP,  TEMP,  Program  Protection  Plan,  and  PESHE 

■  Clinger-Cohan  Act  compliance 
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Enhanced  SE  Summary 


Systems  and  Software  Engineering 


Enhanced  Systems  Engineering  is  the  lynchpin  to  start 
programs  right! 

■  Early  SE  in  support  of  MDD,  MS  A,  and  B 

■  SE  activities  in  support  of  Technical  Reviews  and 
essential  program  planning  efforts 

■  Implementing  SE  ....in  the  right  way 

■  Competitive  Prototyping 

■  Reliability,  Availability,  Maintainability  -  Cost 
implementation 


“Implementing  the  right  activities  at  the  right  time  in  the  right  way” 


Defense  Acquisition  Guidebook  (DAG)  (  http://akss.dau.mil/daq/) 
The  Systems  Engineering  Community  of  Practice 
(https://acc.dau. mil/CommunityBrowser.aspx?id=1 7608); 
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Systems  and  Software  Engineering 


Backup 


22 


RAM  Improvement  Efforts 


Systems  and  Software  Engineering 


■  Some  RAM  Pitfalls  to  avoid  when  executing  a  sound 
systems  engineering  process  include: 

•  Inadequate  planning  for  reliability  and  maintainability 

•  Failure  to  identify  mission  context  or  intended  use 
profile  when  stating  RAM  requirements 

•  Failure  to  design-in  reliability  early 

•  Reliance  on  predictions  instead  of  design  analysis 

•  Inadequate  lover  level  testing 

•  Lack  of  proper  planning,  managing,  and  executing 
reliability  growth  activities,  and 

•  Lack  of  reliability  incentives 
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SE  Provides  a  Technical 
Foundation  for  Acquisition 


Systems  and  Software  Engineering 


MDD 

Business  i 
Decisions  Agreement 
to  pursue 
a  material 
solution 


Engineering 

Support 


c 

CD 

■c 

CD 

O 

c 

3 


Preferred 

System 

Analysis 


MS 
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Selection 
of  a 

preferred 

solution 


Preferred 

System 

Concept 


MS 

B 


Technology 

Maturation 

And 

Prototyping 


Formal 

Program 

Start 


System  i 
Level 
Specs 


Preliminary 

Design 


Completed 
Design 


Material  Solution  Analysis 


Technology  Development 


National 

Research 

Council 


“Pre-Milestone 
A  and  Early- 
Phase 
Systems 
Engineering” 
Jan  2008 


Systems  Engineering  is  most  effective  when  it 
initiated  early  to  start  a  program  right! 
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RAM-C  Activities 

FY  08  FY  09  FY 10  FY11  FY12  FY13  FY 14  FY15  FY16 


I  I DT  QOT  A  Software  Build/LRIP  Delivery  A  Tech  Reviews  A  Program  Reviews  ^Technology  Readiness  Assessment 


Pre-Milestone  B  Sustainment  Requirement  Process 
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Support  Update  to  AoA 
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Trades 


Input 


Draft  CDD 


T  rades 


-Develop  Product  Support  Strategy/Plan 
-Develop  System  Performance  Specification 
-Develop  Lif^  Cycle  Cost  Analysis 
verified  with/analysis,  tests  (Tech  Demos),  or 
comparison/ 

-Perform  CAIV  Analysis 
-Develop  CFP/TPMs 
-Conduct  feasibility  analysis 
-Manufacturing  risks  identified/assessed_ 

-Manage  prototype  development  state  of  the  Art  Ana|ysis 

~h 

I 
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\  \ - ►( 
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SEP 

Develop 

TEMP 


Develop 

APB 


J8 

Provide  APB  I nfrut 
(JROC  InterekProgram)/" 
/ 
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Update  AoA 


pdated— p 
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i 

►  ' 

aRAM-C 

Rationale 

Validate 
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CDD 

-Oversight/Review  Systems  Engineering  Activities 
-Oversight/Review  Program  Documentation 


Approve  Input 
TEMP 


OD/PA&E 
Independent 
Assessment  for 
Milestone 
Decision 


Provide  FD/SC 
Input 


Provide  TEMP  Input 

Evaluate  Prototypes 


Applying 

Business  Process  Modeling 

to 

Develop 

Systems  Engineering  Guidance 

for 

New  DoD  Acquisition  Regulations 

NDIA  Systems  Engineering  Conference 
San  Diego  -  October  2008 


Dr.  Judith  Dahmann 
Aumber  Bhatti 
The  MITRE  Corporation 


Background 


Recognized  need  for  enhanced  SE  early  in  the  acquisition 
process  to  provide  robust  technical  foundation  for 
acquisition  success 

DoD  acquisition  regulations  (DoD  5000)  changes  address 
more  structure  in  early  phases  of  acquisition 

Defense  Acquisition  Guide  (DAG)  updates  to  address  the 
changes  in  acquisition  regulation 

A  Business  Process  Model  of  DoD  5000  and  SE  Guidance 
has  been  constructed  to  provide  technical  support  to  this 
process 


Acquisition  is  a  complex  process  requiring  systems 
thinking  and  SE  analysis  like  other  complex  systems 


DoD  Acquisition  Regulations  and  Guidance 


Regulations 
DoDI  5000.02 


Guidance 
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=\ 

IOC 

FOC 

Materiel 

Solution 

Analysis 

Technology 

Development 

Engineering  and 
Manufacturing 
Development  & 
Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

^  Materiel 
y  Development 
y  Decision 

/\Post-CDR 

Assessment 

LRIP/IOT&E  A  Decision 

v  Review 

^Pre-Systems  Acquisition/  \ 

Decision  Point  /\=  Milestone  Review 


Sustainment  / 


Systems  Acquisition 


Ch 

Topics 

1 

Decision  Support  Systems 

2 

Acquisition  Strategy 

3 

Affordability  &  Life-Cycle  Estimates 

4 

Systems  Engineering 

5 

Life  Cycle  Logistics 

6 

Human  Systems  Integration 

7 

IT  &  NSS 

8 

Intelligence 

9 

Test  &  Evaluation 

10 

Assessments  and  Reporting 

11 

Program  Management 

Focus  of 

current 

activity 


Context  is  worth  50  IQ  Points 
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Draft  Early  Acquisition  Policy  Changes* 


Mandatory 
competing 
prototypes 
before  MS  B 


4 


Coordination  Draft,  DoDI  5000.02 


Why  is  this  hard? 

•  Very  little  experience  with  current  pre-  Milestone  B  SE 
guidance 

-  Makes  it  difficult  to  know  what  to  ‘adjust’  given  changes 

•  The  current  DAG  guidance  is  voluminous 

-  Online  resource  with  over  500  printed  pages  of  information 
without  hotlinks 

•  Limited  understanding  about  the  interdependencies 
among  the  guidance  provided  to  the  program  office  from 
different  perspectives 

-  Any  added  SE  guidance  will  compete  attention  from  already  over 
burdened  program  office 

•  Consequently,  it  was  important  to  understand  how  SE  fits 
into  the  context  of  early  acquisition 

-  What  is  the  relationship  between  SE  and  guidance  for  other  areas 


Need  a  structured  approach  to  understanding  how 

SE  fits  into  larger  context 
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Why  Business  Process  Modeling? 

•  Business  process  modeling  (BPM)  rapidly  articulates  processes 
and  relationships 

-  Supports  communication  and  common  understanding  among  stakeholders 

-  Provides  a  means  for  understanding  relationships  among  concurrent 
stakeholder  activities 

•  Information  to  update  the  DAG  is  closely  aligned  to 
information  for  the  pilot  model;  efficient  leveraging  of  effort 

•  Objective  is  to  support  understanding  of  how  SE  fits  into  the 
larger  context  of  DoD  5000  and  guidance 

•  An  BPM  model  has  been  developed  to  address  SE  guidance  in 
context  of  regulations  and  other  guidance  ‘lanes’  addressing 

-  Proposed  DoD  5000 

-  SE  guidance  (draft  updates  to  DAG  Chapter  4) 

-  Relationships  between  SE  guidance  and  5000  and  guidance  in  other  DAG 
chapters  (limited) 


Model  provides  a  framework  to  articulate  the  role  and 

relationship  of  early  SE 
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Approach 


Iterative  approach  to  building,  reviewing, 
applying  tne  model 

-  Begin  with  a  ‘first  pass’  rapid  development  based  on  the 
current  5000  documentation  using  ‘surrogate’  subject 
matter  expert  (SME) 

-  Review  ‘first  pass’  model  with  SMEs 

-  Update  (second  pass),  review  and  revise 

-  Conduct  an  initial  assessment,  review  and  revise  in 
collaboration  with  stakeholders 

Use  model  as  a  framework  for  enterprise  level 
exchanges 


Version  1 .0  if  the  model  is  in  place  and  in  use 

Work  in  progress 


Notional  Initial  Model  Layout 


o 

o 

o 

in 


AoA  AoA 

Planning  j  Conduct 

AoA  Review 
|  &  MS  A  Prep 

Prototype 

Planning 

IPrototype 
j  Conduct 

Prototypi 

Assessmei 

a  i  Preliminary  !l 
it  |  Design 

3rep  for 
MS  B 

Development 

1 

PDR 

CDR 

JCIDS 

1  \ 

1  1 

1 

Represent  the  SI 

E  actions,  technical  r 

eviews, 

and  products  based  on  ESEWG  drafts 

t  1 

r 

Decision!  Support  Systems 

▼  ! 

Acquisition  Strategy 

Affordability 

Life  Cycle  Logistics 

J 

r 

r 

Human  Systems  Integration 

IT  &  NSSj 

Intelligerice 

Test  &  Evaluation 

T 

[Assessments  and  Reporting 

~r 

Program  Management 

Other  j 

— t 

CO 


U 

CO 

"a 

o 


Birdseye  View  of  the  Model 


Model  provides  a  way  to  visualize  MDD  to  MS  B 
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Results  (i  of  2) 


•  Clear  description  of 

-  Key  elements  of  new  DoD  5000 

•  Provided  a  abstracted  view  of  complex  process 

•  Understand  and  communicate  the  changes 

-  Relationship  among  the  guidance  across  the  DAG  chapters 
particularly  with  respect  to  systems  engineering 

•  Identified  activities  at  different  points  in  the  process 

•  Helped  to  frame  questions  about  relationships 

-  Focus  for  SE  Guidance  during  early  phases  of  acquisition  process 
including 

•  SE  actions  during  each  phase 

•  Expected  input  from  other  processes 

•  Expected  outputs  to  other  processes 

•  Time  criticality  of  information  exchanges 


Model  provides  a  framework  to  look  at  issues  across 

various  guidance  lanes 
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Results  (2  of  2) 


•  Provides  a  framework  for  ‘enterprise’  discussion 

-  Showing  the  numerous  guidance  ‘lanes’  and  where  they 
provide  guidance  to  an  acquisition  program 

-  Identifying  issues  in  aligning  guidance  with  changes  in  policy 

-  Establishing  SE  relationships  with  other  guidance  ‘lanes’ 

•  Identifying  and  managing  interrelationships 

•  Understanding  the  need  and  timing  for  information  sharing 
across  ‘lanes’ 

-  Demonstrating  SE  contributions  to  acquisition  process  and 
work  in  other  lanes 

•  Measuring  the  impact  of  earlier  interactions 

•  Contributing  to  knowledge  base  of  all  ‘lanes’  throughout  the 
process 


Model  provides  a  framework  to  articulate  the  role  and 

contributions  of  early  SE 
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Example:  Best  Practices  for  MDD  to  MS  A 


MDD  to 
MSA 
Slice 


Upfront 
Engineering 
Analysis 


5000 


-,SE  informs 
Decision 
Documents 


Provided  basis  for  DAG  SE  guidance  on 
•  Key  SE  Activities 
Impact  on  program  planning 


JCIDS 


ICD- 


Government 
Program  Office 
Systems 
Engineering 


Other 
Government 
Program  Office 
Activities 


•  Critical  role  for  early  program  office  SE 

•  Advise  and  review  AoA 

•  Engineering  analysis  of  recommended 
solution  forTDS  technical  planning 


AoA 

Guidance 


Key  SE  Activities,  Events  and  Products  and  Their 

Support  to  Program  Planning 
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Example:  Moving  Milestone  B  to  follow  PDR 


PDR  has  been  an  SE  event;  change  impacts  a  range  of  considerations 
outside  of  SE 


MS  B 


Inputs 


►  User  Rqts  (CDD) 

*  System  costs 

*  Test  considerations 

*  Program  protection  issues 

►  Mature  technologies 


Initial  Design 
of  End  Item 


PDR 


Allocate 
systems  to 
subsystems 
Sub  system 
specification  t 


Products 


System  level _ 

requirements 

MS  B 
Products 

Inputs  needed  to  support  design  of  the  end  item 
are  now  produced  as  part  of  the  MS  B  review 


Preliminary 
Design  PDR 


DOD  5000.  Zip"-1 

m-  — 

JCIDst 


Chapti 


Remaii 

Chap 


Knowledge 
To  Inform 
Design 


Preliminary 
Design  ‘Slice’ 
of  Model 


Design 
Knowledge 
To  Inform 
MS  B 


Inputs  MS  B 


Products 


■  Model  provided  a  framework  for  enterprise 
level  discussion 

■  Identified  key  inputs  needed  prior  to 
preliminary  design  including 

■  User  requirements,  cost  constraints,  critical 
technologies,  critical  protection  items 


Topic  of  a  July  workshop  to  address  the 
impact  of  the  change  across  the  guidance 
lanes  (e.g.  DAG  Chapters) 


In  Sum.... 


•  Use  of  BPM  as  a  tool  for  examining  acquisition  policy  and 
guidance  demonstrated  the  value  of  systems  thinking  and 
structured  analysis  of  what  is  in  effect  a  complex  system 

•  Follow-on  possibilities 

-  Extend  model  to  expand  description  of  other  lanes  and  their 
interrelationships,  or  add  other  concurrent  activity  (e.g.  OSD 
oversight  activities) 

-  Animate  model  to  understand  concurrency,  dynamics,  and 
synchronization 

-  Add  notional  resources  (manpower,  time)  for  analysis 

-  Extend  to  focus  on  information  as  a  basis  for  streamlining 
‘documentation’  across  the  acquisition  process 

-  Others.... 


Model  provides  a  framework  for  examining  issues  within  SE  and 

between  SE  and  other  aspects  of  acquisition 


Backup 
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Initial  Model  Scope  Concept:  Focus  on  Early  SE 


Decision 

Lanes 


First  phases  of  acquisition  process, 
subdivided  into  discrete  stages 


CDR 


Represent  the  SE  actions,  technical  reviews,  and  products  based  on  ESEWG  drafts 


Ch  1 
Ch  2 
CD  Ch  3 

u 

c  Ch  5 

co 

"Q  Ch  6 
5  Ch  7 

Ch  9 
Ch  10 
Ch  11 


BIPHBhB  •  1 

/V:;:  1 

T 

Decision  Support  Systems 

▼ 

- 

- 

◄ - 

Acquisition  Strategy 

— 

- : 

— 

1 - 

Affordability 

— 

! 

i 

— 

1 - 

Life  CyclO  Logistics 

T  ] 

♦ 

Human  Systems  Integration 

IT  &  NSS] 

Intelligence 

Test  &  Evaluation 

1 

1  . 

Program 

Management 

Other 

\ 


Include  ‘other  category  for  unknowns 


Represent 
5000  and 
DAG 

Chapter  4 
in  some 
detail 

Represent  SE 
‘inputs’  and 
outputs 

Initially 
represent 
other 
guidance 
‘lanes’  as  SE 
sources  and 
sinks  16 


Birdseye  View  of  the  Model 


Progress  is  being  made  in  developing  DAG  chapter  4 


MSA 


MS  B 


i 


Best  viewed  as  4’  x  10’  version 


Pilot  effort  has  been  initiated  to  explore  use  of  business  process 
modeling  to  examine  relationship  and  alignment  of  regulations  and  SE 

guidance 


ACQUISITION  &  TECHNOLOGY 

-  THE  WILL  TO  CHANGE  - 


Deputy  Director,  Software  Engineering  and  System  Assurance  (SSA) 
Office  of  the  Deputy  Under  Secretary  of  Defense 
(Acquisition  and  Technology) 


October  21,  2008 


NDIA  11th  Systems  Engineering  Conference 

Executive  Panel 


Kristen  Baldwin 


SSE  Way  Forward 


ACQUISITION  &  TECHNOLOGY 

- mmirocHANci - 


★  Reality,  and  opportunity 

★  OSD  SSE  Strategy 

*  Enhanced  SE  Pre-MS  A/B 

*  Human  capital  strategy  for  SE 

*  SE  research 

★  Key  SSE  Improvement  Areas 


eviti 

\  i 

V . 

^ - 

■j-T 

l 

l 

■ : 

☆ 

jy  Key  w 
initiatives 


*  n 


£ 


Institutionalize  SE 


2003 


2008 


2010 


201 


r. 


CoS  &  ffc 


ACQUISITION  &  TECHNOLOGY 

-  TMWtLTOCHANCt  - 


Reality  and  the  Opportunity 


Acquisition  cost 
growth  over  11  years*: 

*  Estimation  changes: 
$201 B 

*  Engineering  changes: 
$147B 

*  Schedule  changes: 
$70B 

*SAR  data  FY  1995-2005 


FY  2008  Defense  Budget 
Total  Obligational  Authority  ($  in  billions)* 


164.7 


116. 


101.7 


75.1 


2.4 


□  MilPers 

□  O&M 

□  Procurement 

□  RDT&E 

□  Revolving  &  Mgt 


^National  Defense  Budget  for  FY  2008  (aka  Green  Book),  March  2007,  page  29. 


With  72%  of  O&S  costs  established  pre-Milestone  A,  Systems 
Engineering  plays  a  critical  role  ensuring  capabilities  are  translated 
into  executable  requirements  and  feasible  programs 


rffil  OSD  SE  Stratc 


>4  ro  o£2S 


•  Treat  Milestones  A/B 

73 

CD 

as  critical;  ensure 

O 

o  z 

completion 

3  73 

3  D 

•  Deliver  needed  #  of 

3  w 

3  £ 

trained  SEs 

Q.  C 

D)  Q. 

•  Perform  pre-MS  A  anal} 

2:  < 
o 

include  stakeholders 

3 

(/> 

•  Implement  Component 

development  planning 

Implement  Achievable 
Acquisition  Strategy  and 
Planning 

Enhance  Gate  Review 
Decision  Process 
Enhance  Staff 
Capabilities 


*Based  on  3700  Program  Assessment  findings 
from  40  Programs  Support  Reviews 


ACQUISITION  &  TECHNOLOGY 

-  THtmi  TOCHANCl  - 


M 


ACQUISITION  &  TECHNOLOGY 

- TMWtLTOCHANCi  - 


jk/J  Enhanced  Systems  Engineering 

- - - 


Policy  and  Guidance  Updates 

*  DoD  5000  update 

*  Acquisition  Guidance  Model 

*  Early  SE  engagement  with  programs 

a  Program  Support  Reviews  (PSRs)  Pre-MS  A/B 

*  Risk  Reduction  activities  (e.g.  Technical  Risk 
assessment  in  AoAs,  Competitive  Prototyping) 

*  SE  Technical  Reviews  -  Informed  Trades  for  Feasible 
Solutions 

Developmental  Test  &  Evaluation 

*  Integrated  DT/OT 

*  Updated  T&E  Strategy  at  MS  A 


TO  0*5 


Systems  Engineering  is  effective  when 
it  informs,  and  is  informed  by, 
other  Acquisition  process  owners. 


ACQUISITION  &  TECHNOLOGY 

- nawniro  CHANGE - 


★ 

★ 


SE  Human  Capital  Strategy 


SE  core  competency  assessment  effort;  completion  - 
Spring  2009 

Program  Systems  Engineer  career  path 

FY08  NDAA  Section  852:  DoD  Acquisition  Workforce 
Development  Fund  -  $300M  per  year  across  DoD 

*  SE  and  T&E  initiatives  to  recruit,  retain  and  train  the 
workforce 

DoD  Human  Capital  Initiative  -  Published  Annex  for 
SPRDE,  PQM  and  T&E 

Partnership  with  INCOSE  SE  Certification  Program 

*  Aligned  with  Defense  Acquisition  Guidance 
Software  Engineering  (SwE)  Human  Capital  Initiatives 

*  DoD  Acquisition  Workforce  SwE  Competencies 
^  Graduate  SwE  reference  curriculum 
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s 


g] 
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Systems  Engineering  Research 


Awarded  SE  Research  UARC 

*  University  Affiliated  Research  Center  (UARC) 

*  Led  by  Stevens  Institute  of  Technology  and  its 
principal  partner,  University  of  Southern  California 

SSE  and  NSA  UARC  Funds 

*  Lead,  coordinate,  and  harmonize  SE  research 

*  Improve  SE  methods,  processes,  and  tools  (MPTs)  in 
support  of  DoD  challenges 

Opportunity  for  DoD  and  Industry  investment 

*  Advance  the  state  of  Systems  Engineering 

*  Nurture  and  grow  graduate-level  systems  engineering 
academic  and  research  programs 


IH> 


qM  &  Tfc 


Key  OSD  SE  Improvement  Areas 


Transcending  DoD  Acquisition 

System/Software  Engineering  Integration 

*  Framework  to  highlight  key  process,  workforce,  and  tools  to 
recognize  key  role  software  plays  in  our  systems 

Systems  of  Systems  Engineering 

*  DoD  SoS  SE  Guide  defines  core  elements  of  SoS  SE,  application 
of  SE  processes,  and  emerging  principals 

Manufacturing  and  Reliability 

*  OSD  and  Component  implementation  of  reliability  best  practices, 
throughout  the  lifecycle  -  July  08  Policy  Memo 

*  Assessing  Manufacturing  Risk  &  Readiness  at  key  decision  points 

System  Assurance  and  Program  Protection 

*  NDIA  Engineering  for  Assurance  Guidebook  integrates  security 
into  Systems  Engineering  to  focus  on  protecting  our  programs 
from  malicious  tampering  and  network  threats 


Always  Our  Focus 


ACQUISITION  &  TECHNOLOGY 

- m  wm.ro  CHANci - 


The  Mission: 

Delivering  Timely  and  Affordable  Capabilities  to  the  Warfighter 


The  Defense  Acquisition  Community 

126,033  Government  and  Military  Certified  Professionals 
500,000+  Defense  Industry  Personnel 


LOGY 


qM  &  Tfc 


^  „  For  More  Information:  Tues  Afternoon  1 


Tuesday,  October  21,  2008 

Session  C  -  1:30-3:15pm 

Track  1  SE  Effectiveness  -  Bayview  III 

7099  DoD’s  Systems  and  Software  Engineering  Revitalization  Efforts — An  Update,  Nicholas  (Nic)  Torelli 

7153  Systems  Engineering  Plan  (SEP)  and  Systems  Engineering  Management  Plan  (SEMP)  Unification, 

Chet  Bracuto 

Track  2  T&E  in  SE  -  Bayview  II 

7100  Implementation  of  the  2007  Developmental  Test  &  Evaluation  Defense  Science  Board  Results,  Chris 

Di  Petto 

7101  Test  and  Evaluation  Value  Metrics  at  Acquisition  Decision  Points,  Darlene  Mosser-Kerner 

Track  3  Program  Management  -  Bayview  1 

7096  New  Acquisition  Policy  and  Its  Impact  on  Defense  Systems  Engineering,  Sharon  Vannucci 

Track  5  M&S  -  Mission  II 

7172  Execution  of  the  Acquisition  M&S  Master  Plan  -  A  Progress  Report,  James  Hollenbach  &  Michael 
Truelove 

Track  8  Software  -  Palm  II 

7137  DoD  Software  Engineering  and  System  Assurance,  Kristen  Baldwin 
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ACQUISITION  &  TECHNOLOGY 


qM  & 
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For  More  Information:  Wed  Morning 


Wednesday,  October  22,  2008 


Session  A  -  8-9:45am 

Track  3  Program  Management  -  Bayview  I 

7438  The  Incremental  Commitment  Model  and  Competitive  Prototyping,  Dr.  Barry  Boehm 

Track  4  Program  Management  -  Mission  I 

7721  Systemic  Analysis  and  Developing  System  Issues,  Peter  Nolte 
7720  Systemic  Root  Cause  Task  Group  Results,  Dave  Castellano 
Systemic  Root  Cause  Task  Group  Recommendations  Implementation,  Nicholas  Torelli 


Track  8  Software  -  Palm  II 

7114  Building  the  Next  Generation  of  Software  Engineers  -  Benchmarking  Graduate  Education, 

Dr.  Art  Pyster 

7135  Improving  Work  Breakdown  Structure  (WBS)  Guidance  for  Weapons  Systems  with  Substantial 
Software  Content,  Christopher  Miller 

Session  B  -  10:15am-Noon 

Track  1  SE  Effectiveness  -  Bayview 

7436  A  Process  Decision  Table  for  Integrated  Systems  and  Software  Engineering,  Dr.  Barry  Boehm 
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For  More  Information:  Wed  Afternoon 


Session  C  - 1:30-3: 15pm 

Track  1  SE  Effectiveness  -  Bayview  III 

6878  Reduction  of  Total  Ownership  Costs  (R-TOC)  and  Value  Engineering  in  the  Defense  System’s  Life 

Cycle,  Chet  Bracuto  &  Dr.  Danny  Reed 

Track  2  Best  Practices  &  Standardization  -  Bayview  II 

6888  Value  Engineering:  Enhance  DMSMS  Solutions,  Dr.  Jay  Mandelbaum 

7761  Applying  Business  Process  Modeling  to  Develop  Systems  Engineering  Guidance  for  New  DoD 
Acquisition  Regulations,  Dr.  Judith  Dahmann 

Track  3  Program  Management  -  Bayview  1 

7344  Complex  System  Development  Program  Assessments  and  Support:  A  Forensics  Perspective,  Dr. 

Dinesh  Verma 

Session  D  -  3:30-5: 15pm 

Track  1  SE  Effectiveness  -  Bayview  III 

7204  Advancing  Systems  Engineering  Practice  within  the  Department  of  Defense:  Overview  of  DoD’s 

Newest  University  Affiliated  Research  Center  (UARC),  Sharon  Vannucci  &  Dennis  Barnabe 

Track  5  M&S  -  Mission  II 

7174  Virtual  Battlespace  Center  for  Systems  Engineering,  James  Hollenbach 


OGY 


qS  &  Tfc 


For  More  Information:  Thurs  Morning 


P6m 


Thursday,  October  23,  2008 


Session  A  -  8-9:45  am 


Track  1  SE  Effectiveness  -  Bayview  III 


7697  Enhancing  Systems  Engineering  in  the  Department  of  Defense,  Ceasar  Sharper 


Track  2  Best  Practices  &  Standardization  -  Bayview  II 


7 1 79  Integration  of  Systems  and  Software  Engineering:  Implications  from  Standards  and  Models  Applied  to 
DoDs’ Acquisition  Programs,  Donald  Gantzer 


Session  B  -  10:15am-Noon 


Track  5  Education  &  Training  -  Mission  II 


7094  Development  and  Validation  of  a  Systems  Engineering  Competency  Model,  Dr.  Don  Gelosh 
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►  as*; 


Improve  Knowledge  through  Technical  Foundation 


Systems  Engineering  is  effective  when  it  informs, 
and  is  informed  by,  other  Acquisition  process  owners. 


Emerging  Contaminants  (EC)  Directorate 


www.denix.osd.mil/MERIT 


Maintaining  Strategic  Advantage 
by  Learning  to  Surf  in  San  Diego 

Your  Surfing  Instructor  is 
Shannon  E.  Cunniff 
Director,  Emerging  Contaminants 
ODUSD  (l&E) 


NDIA  Systems  Engineering  Conference  Oct  2008 


**  **  **  ** 


Emerging  Contaminants  (EC)  Directorate 


www.denix.osd.mil/MERIT 


Today’s  Surfing  Lesson 


Understand  the  Ocean 
Read  Today’s  Conditions 
Proactively  Paddle  or 
Miss  the  Wave 
Sustain  your  Ride! 


Emerging  Contaminants  (EC)  Directorate 


www.denix.osd.mil/MERIT 


Lesson  One 


Understand  the  Ocean 


Emerging  Contaminants  (EC)  Directorate 


www.denix.osd.mil/MERIT 


Trends 

♦  Economic  strength  /  growth 

»  Energy  costs  increasing 

»  Environmental  liabilities 

»  Discretionary  spending 
shrinking 

»  Frustration  with  ATL  spending  & 


♦Legal 

»Regulations,  Treaties 
»EO 13423 

»Regional  Agreements 


Evolve  to  remain  relevant  and  ready  to  meet  these  challenges. 


Emerging  Contaminants  (EC)  Directorate 


www.denix.osd.mil/MERIT 


Progression  of  Environmental  Practice 


Total  Life-Cycle 
Assessment 


Pre 

1970 


1 980-90S 


2000s  2007... 


**  ■*  ■#  ■* 


Emerging  Contaminants  (EC)  Directorate 


www.denix.osd.mil/MERIT 


National  Chemical  Risk  Mgmt  Trends 

Use  of  Precautionary  Principle 

»  Must  understand  health  &  environmental  effects  before  using  chemicals 

Chemical  Management  and  Green  Chemistry 

»  E.U.  REACH,  EO  13423,  U.S.  ChAMP,  likely  Toxic  Substances  Control  Act 
reforms 

Biomonitoring  -  What’s  showing  up  in  humans? 

»  Center  for  Disease  Control’s  national  biomonitoring  &  Calif,  voluntary 
program 

Evolving  Risk  Assessment  Process 

»  Increasing  transparency... showing  uncertainty  range 
»  Identifying  science  gaps  early  and  filling  gaps  via  research 
»  Shift  from  animal  dose/response  — 4oxicogenomics  with  human  cells 
»  Use  of  computational  sciences 

»  Application  of  Age-Dependent  Adjustment  Factor  (ADAF) 


■>  •>  ■>  a* 
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RoHS  and  Lead  -  A  Cautionary  Tale 

(continued) 

One  RoHS  Goal:  Eliminate  Lead  from  Electronics 

Aeronautical/Aerospace  Applications  Constitute  ~  1% 
of  Worldwide  Electronics  Usage 

♦  DoD  a  fraction 

Lead-free  Circuit  Boards  Are  In  Our  Supply  Chain 

♦  Where?  What  is  the  impact  on  mission-critical  applications? 

Initiatives  Underway  at  DoD  to  Address  These 
Unintended  Consequences 

♦  All  are  expensive  (time-consuming) 

♦  All  are  re-active  (vice  pro-active) 
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Lesson  Two 


Read  Today’s  Conditions 


*  *  *  * 
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REACH  -  Basic  Background 

Main  Objectives 

♦  Reduce  risk  from  chemicals 

♦  Share  information  on  chemicals  affects 

♦  Encourage  substitution  to  safer  substances 

♦  Authorize  or  restrict  the  use  of  high  concern  chemicals 

♦  2009-2018  Progressive  implementation  based  on  quantity  &  hazard 

Directly  Affects 

♦  Importers  to  EU  &  EU  based  manufacturers  to  be  responsible  for 
assessing  the  health  and  environmental  effects  of  every  substance 

♦  Importers  to  EU  &  EU  based  manufacturers  to  transmit  information  to 
downstream  users 

♦  Downstream  users  to  apply  risk  management  measures 


**  •# 
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REACH  -  Basic  Background  (con’t) 

Requires  Manufacturers  and  Importers  to  Register 
Listed  Chemicals,  which  Raises  Issues  About: 

♦  DoD’s  status  and  role  are  complex  and  unclear 

♦  Impact  to  DoD’s  suppliers  (both  in  and  into  EU) 

♦  Proprietary,  business  confidential  and  national  security  info 

First  Impacts  to  DoD 

♦  If  by  November  30,  2008,  if  some  party  has  not  registered  those 
High  Production  Chemicals  that  DoD  uses,  its  possible  that  then 
DoD  may  start  feeling  the  effects  of  REACH. 
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REACH:  Generating  Emerging 
Contaminants  for  the  Next  10  Years 


What  is  an  Emerging 
Contaminant? 


*:  Chemicals  &  materials  with 

♦  Perceived  or  real  threat  to 
human  health  or  environment 

♦  Either  no  peer  reviewed  health 
standard  or  an  evolving 
standard 

*:  May  have 

♦  Insufficient  human  health 
data/science 

♦  New  detection  limits 

♦  New  exposure  pathways 
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Defense  Exemptions  are  Possible ,  Not  Guaranteed 

*:  Specific  cases... certain  substances... necessary... Defense. 

*:  Treaty  of  Lisbon  2007-  EU  greater  say  on  Defense  matters 
*:  Not  EU-wide  --  Country  by  Country  Exemption  --  30  Countries 
•:  Labor  intensive  to  get 

•:  Likely  to  Require  Some  Proof  of  Military  Uniqueness  and  Lack  of 
Substitutes 

•:  US  not  an  EU  Member  State 

♦  DoD  not  obligated  to  comply  with  EU  laws 

♦  Sovereignty  issues 

♦  SoFA  /  Bi-Lateral  agreements 

*:  However,  for  EU  Nations 

♦  Compliance  is  mandatory 

♦  May  be  subject  to  sanctions  for  non-enforcement  within  their  borders 


*  *  *  *  *  *  *  *  A  *  *  *  ■ 
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DoD  &  Defense  Industries:  Stormy  Seas? 

Potential  for  Release  of  Sensitive  Information 

♦  Required  disclosures  could  reveal  sensitive  material  formulations 

Foreign  Military  Sales 

♦  US  may  not  have  access  to  needed  maintenance  or  logistic  supports  in  EU 

♦  Competitive  advantage  to  EU  if  US  suppliers  do  not  comply 

NATO  Interoperability/Unknown  Performance  Factors 

♦  EU  military  may  not  be  able  to  use  US  systems,  maintenance  procedures,  or  logistic 
supports 

Overseas  Maintenance  and  Base  Operations 

♦  Chemicals  required  by  for  maintenance  may  not  be  available 

♦  May  not  be  able  to  import  articles  made  with  or  containing  some  chemicals 

Cost  and  Availability 

♦  Diverging  defense  &  commercial  sectors:  Possible  problems  with  availability  of  parts 
and  materials 

Compatibility  Issues  &  Pressure  to  Expand  Qualified  Products 
Lists 

♦  RDT&E  of  substitutes  --  alternatives  that  meet  military  specs 

♦  Unknowing  acceptance  of  alternatives 

Complicated  and  varying  MOD  requirements  for  Defense 
Exemptions 


■  * 
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REACH ...  a  Surfboard? 

Knowledge  Management  Benefits  Other  DoD 
Interests  and  Activities 

♦  Inform  material  selection  to  avoid  late  change  orders 

♦  Lifecycle  cost  reductions 

♦  EO  13423’s  chemical  risk  management  goals 

♦  Strategic  materials  identification  for  National  Defense 
Stockpile  decision  making 
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EU  1st  Round  SVHC  &  DoD  Chemicals  of  Interest 


FOR  IMMEDIATE  CONCERN 

Sodium 

dichromate 

Large  potential  impact  since  it  is  used  in  many  conversion  coatings  and  primers  repaint  of  all  DoD  aircraft  skins, 
although  less  than  first  suspected  on  F-1 6s;  much  will  depend  on  which  products  have  been  qualified.  May  also 
be  used  in  chromate  washes  prior  to  vehicle  painting.  In  many  formulations,  zinc  chromate,  barium  chromate, 
strontium  chromate  or  other  chromates  can  be  used  instead.  <Sodium  dichromate  dihydrate  was  ‘screened’  in 
‘07  because  it  showed  up  on  an  NTP  list.  There  were  57  items  in  HMIRS  most  were  reagent  grade  for  lab  use 
and  a  number  of  photo  developer  cleaning  applications. > 

Cadmium  (Cd)  - 

containing 

products 

Restrictions  on  Cd  use  for  vehicles  come  into  effect  June  1,  2009  (aircraft  exempted  for  now)  includes  fasteners 
and  bolts.  DoD  may  not  be  able  to  obtain  Cd-plated  components,  even  if  allowed  to  use  them;  major  impacts  to 
repair  and  overhaul  can  be  expected  for  trucks,  for  example,  since  few  qualified  alternatives  (ZnNi  plate,  Al 
coatings),  especially  for  fasteners. 

Asbestos 

Used  for  some  turbine  engine  washers,  gaskets.  Existing  items  can  be  used,  but  not  replaced,  with  asbestos. 

OTHER  CHEMICALS/USE  OVERVIEW 

CAS /EC 
Numbers 

Reason 

Recently  Compiled  DoD  Information 

Anthracene  is  used  in  the  manufacture  of  pyrotechnics 
and  as  a  component  of  black  smoke 

120-12-7/ 

204-371-1 

PBT 

May  be  of  concern  since  it  is  used  in  dyes  (flares 
and  markers) 

HMIRS  -  37  products;  MIDAS  -  32  items 

4,4'-Diaminodiphenylmethane  is  used  as  a 
hardener  in  epoxy  resins  and  adhesives  as  well  as  in 
some  construction  coatings 

101-77-9/ 

202-974-4 

CMR 

Could  become  a  big  issue  as  DoD  uses  many 
adhesives  (chemistry  to  be  identified) 

HMIRS  -  253  products,  curing  and  hardening 
agents,  adhesive  film 

Cobalt  dichloride’s  widespread  uses  include  the 
production  gas  masks,  self  indicating  silica  gels,  flux  for 
magnesium  refining  (notably  when  recycling  scrap 
material),  as  a  solid  lubricant,  a  metal  drier  in  air-drying 
coatings  and  a  drying  agent  in  paints,  lacquers, 
varnishes  and  printing  inks;  in  the  production  of  non- 
ferrous  metals  and  electroplating  processes 

7646-79-9  / 
231-589-4 

CMR 

HMIRS  -  215  products;  MIDAS  -  113  items 
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Lesson  Three 


Proactively  Paddle  or  Miss  the  Wave 


#  *  m #  *  •  #  *  #  * 
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Steps  for  Catching  the  REACH  Wave: 

What  DOD  &  its  Suppliers  Can  Do 

Identify  Strategic  Materials/Chemicals  and  Identify  Needs  for 
Defense  Exemptions  Early 

Coordinate  Research  Plans  to  Look  For  and  Evaluate  Substitutes 
Accelerate  &  Expand  Substitution  Efforts 
Improve  Visibility  into  Supply  Chain 

♦  Materials  used 

♦  Chemicals  required  for  O&M 

Improve  Knowledge  Management  and  Information  Sharing 

♦  E.g.,  Uses  of  proposed  SVHCs  to  ensure  those  uses  authorized 


DoD’s  Emerging  Contaminants  Directorate  Can  Help  You 
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REACH  and  EC 
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EC  “Scan-Watch-Action”  Process 


Over  -the-  horizon 


I 


vvatch 


action 


EC  News 

Possible  DoD  impacts 

Phase  1 

Assessment 

Probable  high  DoD  impacts 

w 

Phase  II 

Assessment 

risk 

management 


Review  literature, 
periodicals,  regulatory 
communications,  etc. 

Monitor  events; 
Conduct  Phase  I 
qualitative  impact 
assessment 

Conduct  Phase  II 
quantitative  impact 
assessment  with  risk 
management  options 


Risk  Management  Options  to 
Governance  Council 


Probability  of  Adverse  Impact 
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Integrated  Risk  Management 


•:  Define  the  negative 
influences  on  the 
enterprise  in 
question. 

•:  Identify  strategic 
risk  management 
options  to  lower 
severe  risks. 

<  Measure  progress 
by  quantifying  risk 
reduction  of  actions 
taken. 


Emerging  Contaminants  (EC)  Directorate 


www.denix.osd.mil/MERIT 


Hex  Chromium  Phase  I  Impact  Assessment 

Completed  July  2007 


Hexavalent  chromium  is  used  in  DoD  weapons  systems  due  to  its  useful  metal  finishing 
properties.  Chromium  compounds  enhance  hardness,  increase  adhesion  as  paint  primers, 
and  provide  corrosion  protection.  Undergoing  IRIS  reassessment  and  CAL/EPA  is  developing 
drinking  water  public  health  goal.  ;  ^ 


Likelihood  of  Toxicity  Value/ 

Regulatory  Change 


1 .  Likelihood  that  the 
USEPA  will  revise  the 
IRIS  toxicity 
benchmarks  for 
Hex  Chrome 

2.  Likelihood  that 
OSHA  will  revise 
the  occupational 
exposure  standards 
for  Hex  Chrome 


Likelihood  Timeframe 


-  5-8  yrs 

-  1-5  yrs 


H 


Q) 

0) 

> 

■O 

< 


■ 

n 

(0 

n 

o 

o. 


L 


x 


► 


Severity  of  Adverse  Impact 


H 


Note:  California  may  establish  a  Public  Health  Goal 

before  USEPA  finalizes  its  IRIS  value  or  reassesses  ES&H  POM  D  of  Assets 

the  federal  MCL.  Readiness  &  Training  X  Cleanup 


Acquisition/RDT&E 
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DoD  Action  List 

•:  Perchlorate 

•i  Royal  Demolition 
explosive  (RDX) 

*5  Trichloroethylene  (TCE) 

•:  Hexavalent  Chromium 

<  Naphthalene 

■4  Beryllium 

•:  Sulfur  Hexafluoride 
(SF6) 


Note:  Some  risk  management 
actions  underway  including 
research  on  toxicity,  substitutes, 
&  treatment. 


DoD  Watch  List 

S  Tungsten 

S  Tetrachloroethylene  (PCE) 

S  Dioxins 
S  1,4-Dioxane 

•  Nanomaterials 

S  Perfluorooctyl  sulfonate  (PFOS) 

S  Di-nitrotoluenes  (DNT) 

✓  Lead  (Added  3-07) 

S  Nickel  (Added  3-07) 

•  Cerium  (Added  7-07) 

•  Cobalt  (Added  7-07) 

•  Cadmium  (Added  12-07) 

•  Manganese  (Added  12-07) 

S  Perfluorooctanoic  acid  (PFOA) 

(Downgraded  from  Action  List  9/08) 


S  -  Phase  I  Impact  assessments  completed 
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The  Products 

(Those  Most  Relevant  to  AT&L  and  Industry) 


ACQUISITION 
TOXICITY  TESTS 


TIER  I -III 
TOXICITY  CRITERIA 


Operations  & 
Support...  & 
Disposal 


Refine  initial  concept, 
Develop  Technology, 
Development  Strategy 


Reduce  technology  risk  and 
determine  appropriate  set  of 
technologies  to  integrate  into  a 
full  system. 


Develop  a  system  or  increment  of 
capability;  reduce  integration  and 
manufacturing  risk;  ensure  operational 
supportability;  reduce  logistics  footprint; 
implement  human  systems  integration; 
design  for  producibilty;  ensure 
affordability  and  protection  of  critical 
program  information;  and  demonstrate 
system  integration,  interoperability, 
safety,  and  utility. 


Achieve  operational  Execute  a  support 

capability  that  satisfies  program  that  meets 

mission  needs.  operational  support 

performance 
requirements  and 
sustains  the  system  in  the 
most  cost-effective 
manner  over  its  total  life 
cycle.  Dispose  of  the 
system  in  the  most  cost- 
effective  manner  at  the 
end  of  its  useful  life. 


Impact  Assessments  and  Risk  Management  Options 
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Emerging  Contaminants  Public  Web  Site: 

https:llwww.denix.osd. millMERIT 


EC  PORTAL:  www.ecportalinfo.org 


Working  on  More  Powerful  Ways  to 
Collect,  Disseminate,  and  Share  Information  &  Experiences 


Basic 

Functional  Areas 

On  These  Lists 

Tracking 

SMEaiUlC 

Effects 

More  Info 


Sulfur  Hexafluoride  (SF6) 


Name: 

You  are  now  logged  tit  as  a  data  administrator 

Basic  Chemical  Information  |  Merit  Status 


Stati  is:  ■  V  a  I  -  :i .  CAS  N  u  m  l>  e  r:  255 1  -62-4  L  ast  U  |>  d  ate  <1 :  3/24/2008 

(mms  drf  yyyy) 


added  to  watch  list 

Action  Date 

Executive  Summary  Introduction 

Sulfur  hexafluoride  (SF6)  is  a  dense,  gaseous  compound  that  is  colorless  and  odorless.  Under  standard  conditions, 
it  is  not  flammable  or  reactive.  Not  particularly  toxic  to  humans,  the  main  health  risk  associated  with  SF6  is  the  risk 
of  asphyxiation  when  in  an  enclosed  space  with  high  concentrations  of  the  gas.  SF6  is  used  in  several  industrial 
and  military  applications;  however,  it  is  extremely  unfriendly  to  the  environment  and  may  be  restricted  or  banned  in 
the  future. 


Why  Emerging? 


SF6  has  the  potential  to  be  included  in  the  Clean  Air  Act  and/or  the  Global  Warming  Pollution  Reduction  Act.  If  this 
happens,  and  the  amount  of  SF6  emissions  is  restricted,  it  could  affect  the  DoD.  SF6  is  used  in  several  military 
applications,  and  as  of  today,  there  are  no  viable  alternatives.  DoD  would  have  to  invest  time  and  resources  to 
continue  development  of  alternatives  and  reduce  emissions  from  existing  sources. 


Done 
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Lesson  Four 


Sustain  Your  Ride 
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Address  Emerging  Contaminants  Early! 

Proactive  vs.  Reactive  Actions 

Health  Impacts 

Cleanup  Costs 

Compliance  Costs 

Readiness  Impacts 

Platform/Facilities  Life 
Cycle  Costs 

$$$ 

Large  Benefits 


Sustainability  Fosters  DoD’s  Mission 


Early  EC  ID  &  Ris 
Management  Actions 


$$$ 

Small  Investment 


wm  «  **  ** 
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Sustainability  Fosters  DoD’s  Mission 

Strengthens  Operational  Capacity 

♦  Meet  current  and  future  training,  testing,  and  other  mission 
requirements  by  sustaining  land,  air,  and  water  resources 

Lessens  Costs 

♦  Minimize  impacts  and  total  ownership  costs  of  systems,  materiel, 
facilities,  and  operations 

Enhances  Well-Being 

♦  Of  our  Soldiers,  civilians,  families,  neighbors  and  communities 

Links  the  Future  to  the  Present 

♦  Fosters  identification  of  user  needs  and  anticipation  of  future 
challenges 
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S  Disciplined  People 

✓  Act  with  Understanding 

✓  Broadened  temporal  & 
areal  scales 

S  Disciplined  Thought 

✓  Broadened  System 
Boundaries 

✓  Risk-based  Approaches 

✓  Life-cycle,  Ownership  of 
the  risk,  Risk  taker  pays 

✓  Moving  beyond 
compliance 

v'  Disciplined  Action 

✓  Greater  Accountability 


^Distinctive  Impact 

^Superior  Performance 
for  the  Mission 

^Enhanced  Endurance 

•s  Strategic  &  Economic 
Advantage 


Sustainability  is  about  Building  Greatness  to  Last 


*  ■  *  *  •  *  *  • 
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Surfing  Lesson  Main  Points 


Understand  the  Ocean:  New  Paradigms  Forcing  Change 

♦  Budget 

♦  Agility  needed  to  maintain  strategic  advantage 

Read  Today’s  Conditions 

♦  REACH  is  just  the  beginning... its  going  to  get  more  complicated  in  a  world 
economy  and  supply  network 


Proactively  Paddle  or  Miss  the  Wave 


♦  Requires  new  thinking:  Proactive  targeted  investments  before  regulatory 
action 

♦  EC  providing  advance  warning  and  tools 

Sustain  your  Ride! 

♦  Potential  large  payback 

♦  Protects  people,  mission  and  assets 
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Take  Home  Message 

Either  stay  ahead  of  the  curve 
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Or  wipe  out 
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Questions  &  Discussion 
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Hexavalent  Chromium 


•:  Hexavalent  chromium  is  a 
metal  that  is  used  for  coatings 
in  aircraft  and  other  vehicles 
to  provide  a  hard,  wear- 
resistant  surface,  and  in 
paints  to  prevent  corrosion  of 
the  base  metal 


_ 


•:  The  Permissible  Exposure 
Limit  (PEL)  was  recently 
lowered  by  the  Occupational 
Safety  and  Health 
Administration  (OSHA) 

•:  European  environmental 
regulations  have  effectively 
banned  the  use  of  hexavalent 
chromium  on  vehicles  and 
electrical  equipment.  Many 
automobile,  military  parts  and 
electronics  manufacturers  are 
adopting  European  or  other 
stringent  standards  for  all  of 
their  products 
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Hex  Chromium  Phase  I  Impact  Assessment 

Findings 

♦  Environment,  Safety,  and  Health  (ES&H) 

♦  High  risk  because  it  is  a  known  inhalation  carcinogen — CrVI  is  also  a  suspected  oral  carcinogen 
that  poses  noncancer  risks.  May  be  more  stringently  regulated  due  to  new  toxicity  testing  results. 
Significant  cost  and  effort  required  to  monitor  and  manage  worker  exposure  if  standards  are 
lowered. 

■  Readiness  and  Training 

♦  Low  risk  due  to  the  possibility  of  reduced  availability  of  ranges/firing  points  as  a  result  of  new 
regulations  is  considered  small. 

A  Acquisition/Research,  Development,  Testing,  and  Evaluation  (RDT&E) 

♦  High  risk  because  over  2,300  munitions  items  contain  CrVI.  Aircraft  demolition  and  shipwrecking 
also  releases  CrVI.  Emerging  regulatory  constraints  may  increase  life-cycle  costs  and  restrict 
testing/development  of  new  technologies. 

•  Production,  Operations,  Maintenance  &  Disposal  (POMD)  of  DoD  Assets 

♦  High  risk  as  new  CrVI  toxicity  values  would  impact  some  routine  anti-corrosion  inspection  and 
painting  processes.  Waste  handling  and  disposal  burdens  would  increase  as  would  permitting 
and  reporting  for  many  DoD  industrial  operations. 

x  Cleanup  Program 

♦  High  risk  as  cleanups  at  200-250  DoD  sites  may  be  affected.  Very  likely  will  have  to  re-examine 
closed  sites  for  possible  re-evaluation. 

>  Recommendation:  Phase  II  Impact  Assessment  in  process/RMOs  under  development. 


**  ** 
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Beryllium 


Beryllium  is  a  steel-gray,  naturally 
occurring  metal  found  in  rock,  coal,  soil 
and  volcanic  ash 

It  is  used  to  make  specialty  ceramics 
for  electrical  and  high-technology 
applications  such  as  x-ray  machines, 
spaceships  and  aircraft,  missile 
guidance  systems,  and  computers 

OSHA’s  exposure  limit  is  2 
micrograms/cubic  meter  of  air.  Under 
the  Clean  Air  Act,  EPA  restricts  the 
amount  of  beryllium  that  can  be 
released  into  the  air 
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Naphthalene 


•:  Naphthalene  is  a  natural  constituent 
of  petroleum  and  jet  fuel  used  by  the 
military.  It  also  appears  as  a  white 
solid  in  pesticides  (e.g.,  mothballs) 

•:  Naphthalene  is  classified  by  the 
National  Toxicology  Program  as 
reasonably  anticipated  to  be  a  human 
carcinogen 

•:  EPA  is  evaluating  potential  regulatory 
changes 

•:  There  are  potentially  significant 
impacts  on  health  and  DoD 
operations,  especially  fuel  handling 

•:  Further  engineering  controls, 
personal  protective  equipment,  air 
monitoring,  and  medical  tracking  may 
follow 


•:  DoD  complies  with  current 
environmental  and  occupational 
health  regulations 

•:  DoD  is  testing  jet  fuel  samples  and 
evaluating  potential  impacts  on  DoD 
related  to  possible  changes  in 
regulatory  status 


» 


*  ■  *  *  M  •  • 


Emerging  Contaminants  (EC)  Directorate 


www.denix.osd.mil/MERIT 


Perchlorate 


Perchlorate  is  a  salt  with  properties  that  make  it  the 
safest,  most  efficient,  stable  and  reliable  propellant 
oxidizer  available 

DoD  relies  on  perchlorate  for  rocket  and  missile 
propellants,  pyrotechnics  and  flares,  but  is  relying 
on  it  less  and  less  for  munitions 

Perchlorate  was  detected  —  generally  at  levels  below 
EPA’s  benchmark  of  24  parts  per  billion  —  in 
drinking  water  sources  in  at  least  34  states 

Several  states  such  as  California  are  considering  or 
have  recently  developed  public  health  goals  or  other 
regulatory  requirements 


Emerging  Contaminants  (EC)  Directorate 


www.denix.osd.mil/MERIT 


Set-up  of 
European 
Chemicals 
Agency 


REACH  -  Timeline  and  Phases 


Registration  of  Substances  of  1  tonne  or  more/year 

Pre-registration  of 
Phase-in 
Substances 

Registration  of  Substances  of  100  tonnes  or  more/year 

r 

Registration  of  Substances  of  1000  tonnes  or 
more/year 

CMR  Substances  of  1  tonne  or  more/year 

N:  R50/53  Substances  of  100  tonnes  or 
more/year 

Registration  of  New  Substances 


1  1 

i 

l 

l 

l 

i 

i 

i 

i 

31 

31 

May 

May 

2013 

2018 

1 

r 


1  June 
2007 
EiF 


December 

2008 


June 
2009 

Restrictions 


■ 


r 


30 

November 

2010 


Focus  First  on  substances  with  hiah  volumes  and  those  of  greatest  concern. 


NDIA  11th  Annual  Systems  Engineering  Conference 


ESTABLISHNG  A 
SYSTEM  OF  SYSTEMS 
SYSTEMS  ENGINEERING 
ORGANIZATION  IN  THE  ARMY 


ROSS  R.  GUCKERT 

Assistant  Deputy  for  Acquisition  and  Systems  Integration 
Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics  and  Technology 

Ross.Guckert@us.army.mil 

21  October  2008 


UNCLASSIFIED 


ASA 

Challenges  for  the  Arm 


No  System  of  Systems  (SoS)  Systems  Engineering  capability  at  the  Enterprise  level 

-  Stove-pipe  product  development 

-  Many  interdependencies 

-  Path  from  Current  to  Future? 

-  SE  critical  to  LandWarNet  Battle  Command  and  operational  GWOT  rotations 

No  “Integrator”  for  Brigade  Combat  Teams  (BCTs)  and  support  Brigades 

Institutionalizing  Reliability  Programs 


Army  systems  are  becoming  more  interdependent  and  required 
operational  capability  is  not  provided  by  a  single  system,  but  rather  a 

combination  of  systems 


UNCLASSIFIED 


ASA 

ASA  (ALT)  SoS  SE  Organization 


MISSION 

Provide  Systems  Engineering  capability  at 
System  of  Systems  level  across  the  Army  enterprise  to 
deliver  integrated  and  interoperable  weapon  systems  that 
provide  optimized  and  affordable  capability 

FUNCTIONS 

Develop,  evolve,  and  maintain  a  detailed,  interoperable  SoS  design  baseline  -  Enterprise  Systems  Architecture 

Address  technical,  operational  and  cost  aspects  to  frame  issues  for  decision  making 

Leverage  experimentation  and  M&S  tools  as  part  of  engineering  analysis/operational  assessment 

Establish  and  evolve  an  SoS  vision  over  time,  and  translate  into  capability  attributes 

Translate  emerging  requirements  into  implied  system  attributes  for  technology  insertion  solutions 

Lead  targeted  technical  assessments  to  enable  cost/capability  trades  within  and  across  system  boundaries 

Maintain  visibility  into  individual  system  architectures,  specifications  &  performance 

Coordinate  technically  with  SEs  in  related  programs  (Army,  Joint) 


UNCLASSIFIED 


ASA 

SoS  SE  and  PEO  Relationship 


COORDINATION/SUPPORT: 

TRADOC 
'  ARSTAFF 
OSD/Joint  Programs 
RDECs 


ASA  ( AL&T)  SoS  Systems  Engineering 

•  Policy 

•  Oversight 

•  Enterprise  level  system  architectures 

•  Enterprise  level  analysis,  evaluations,  trade 
studies  -  End-to-end  performance 

•  Synchronize  enterprise  level  development 

•  Identify  and  resolve  cross-portfolio  issues 


PEO  Portfolio  SoS  Engineering 

Oversight  of  POR 

Portfolio  level  architecture  (to  include  cross-portfolio 
requirements) 

Portfolio  level  analysis,  evaluations,  and  trade  studies 
SoS  responsibilities  -  Works  to  resolve  cross-domain 
issues 

PEO  -  Lead 

RDEC,  FFRDC,  SETA  -  Support 


UNCLASSIFIED 


ASA 

SoS  SE  Governance 


Vision,  direction, 
policy 


SoS  Baseline/ 
Execution  plan / 
Resolution 


Implementation/ 
Execution 


Tier  2 


AAE/MILDEP/DASA(A&SM) 

Evolve  vision,  establish  policy 
Synchronize  w/ARSTAFF  Leadership 
Oversight  &  decisions 

SoS  SE  Org 

Maintain  Enterprise  Systems  Arch 
Facilitate  cross-portfolio  issue  resolution 
SoS  trade  studies/engineering  analysis 
Synchronize  enterprise  level  development 
(LWN,  SWB,  C2  Conv,  UBC) 


Tier  3 


PEOs 

Technical  Execution/Implementation 
SoS  Eng  &  Arch 
Support  trades 


UNCLASSIFIED 

| 

- 


ASA 

Example:  IBCT  Snapshots 


CDR 


CO],  02CO0  (COMMANDER)  P&C 
I.TC  1M00  (FIRE  SPT  COORD)  PAT 
SSG  1 1 B30  I  ASST  OPNS  SGT)  ( 

SGT  1 1  H2< )  (ASST  <  >I*NS  SCT)  C 


ICS 


& 


JTRS 

GMR 


FSCOORD 


JTRS 

HMS 


AN/VRC- 
1 04(V)3 


SRW 


WNWj 


SGRS 


SGRS 


MUOS 


DCDR 


£ 


COL  U2«  00  (DEP  COMMANDER)  P&C 
(  PI  LUOO  (FIRE  (  ONT  OFF)  ( 

SSC  IIB30  (ASST  OPNS  SGT)  C 
PIC  I IIIIO  (V Ell  DRIVERX 


ICS 


JTRS 

GMR 


JTRS 

HMS 


AN/VRC- 

104(V)3 


WNW| 


SRW 


SGRS 


MUOS 


□□ 


2<]tac  i 


ICS 


JTRS 

GMR 


MAJ  JIAOO  (BDE  ENGR  OFrt  P 
|  CPT  l.TWOi  .ASSTSXC 
|  CPT  V'TOO  (INTEL  OID  C 
f  PT  57A00  (BTl.  CMT)  OFTl  f 
|  CPT  748(10  (CKEM  OFTl  C 
j  SGM  MZ-'O  i  OPNS  SGT*  C 
j  SSG  19030  (OPNS  NCO)  C 
|  SSG  7 1830  (RECON  SGT*  C 
SGT  i'B’O  I  SR  INFO  SYS  SPi  C 


JTRS 

HMS 


WNW 


SRW 


SGRS 


SGRS 


MUOS 


CMDCTRSYS 


CPOF  CPOF  MCS  MCS 

MCS  MCS  MCS  DCGS-A 
BAL 


^hhccmd^vhfT^ 


|  HHC  | 

HHC  CDR 


HHC  XO 


<9 


SINCGARS 

90F 

|  SGRS | 

JTRS 

HMS 

!  SRW] 

JV5 


<9 


SINCGARS 

87F 


JTRS 
1  HMS 


[SGRS  | 


SRW 


JV5 


HHC  SUPPLY 


1  SINCGARS 

|  SGRS; 

87F 

TC-AIMS  I 

^  G) 

GCSS-A  V2  UBC  Beacon 
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ASA  (AL&T) 


Synchronization  with 
LandWarNet 


UNCLASSIFIED 


LWN/BC  Capability  Set  Management  Process 


“Capability  Set  Life-Cycle” 


DEFINE  &  DEVELOP 


What  can  be  provided  when  at  affordable  price? 

•  SoS  Engineering  Analysis/Trades 

•  SoS  Synchronization 

•  Technical  Feasibility 

•  Inform  decisions 
“Bang  for  the  Buck” 


CABILITY  NEEDS  ANALYSIS 


PRIORITIZE 

^^BILITY 


Does  Capability  Set  stand  up  to  Oper  Analysis? 

■  Exercise  Cap  Set  through  Oper  Analysis  - 

leverage  analytic  tool  suite 

■  Adjust  to  changes  (funding,  rqmt,  force  changes,  etc.) 

■  Assess  changes  on  SoS  perf  &  synchronization 
•  Re-assess  “Bang  for  the  Buck” 


ESTABLISH 

CAPABILITY 

SET 

PARAMETERS 
(OPN,  TECH,  FISCAL) 


STEP  2 

INTEGRATE 

ARCHITECTURES 

APABILITY  SEGMENTS 

STEP3| 


STEP  4 


Fiscal  Analysis 


R-7  Years 
REFINE 


SCREEN  &  ID  SOLUTION  SET 


APPROVE 


'MHrk 

SELECT 

CAPABILITY  SET 
COA 


LWN  GOSC 

R-36  months 


IT  FOR  CHANGE  ENVIRONMENT 
?W  TECHNOLOGY,  ONS/JUONS,  FORCE  SIZING) 


PRODUCE/ 

PRIORITIZE 

COAS 


SOSE  ANALYSIS  OF 
CAPABILITY  SETS 


Understood  Operational  Effects  Through  Operational  Analysis  (M&S) 


STEP5| 


YnWrfr 

Approve 

“Baseline” 

CAP.  SET  15-16  FOR 
REFINEMENT 

LWN  GOSC 

R-6  Years 


Develop 
“BASELINE” 
INTEGRATED 
CAPABILITY 
SET 


SYNCH 


BO  IP  Lock 

?0%  So/tri/on' 


Near  Term  Trades 


APPROVE 

Final 

CAPABILITY  SET 
Synched  wARFORGEN 

LWN  GOSC 

R-18  Months 


“Good  Idea” 
Cut-Off 


ONS/JUONS 


> 


STEP7| 


FIELD 

ARFORGEN  RESET 
SCHEDULE 


SYNCH 


CAPABILITY 

SET 

Testing  &  Certification 


MTOE  Lock 


Force  Validation  Conference 
Army  Sourcing  Conferences 
Army  Equipping  Conferences 


ASA  (AL&T) 


Army  Reliability  Initiatives 
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UNCLASSIFIED 


ASA  (AL&T)  — 

Army  Reliability  Policy 


Mandates  development  and  demonstration  of  a  mid-SDD  reliability  test 
threshold  for  all  pre-Milestone  B  programs  with  a  JPD  of  JROC  Interest1: 

-  Default  value  is  70%  of  CDD  reliability  requirement 

-  Must  be  demonstrated  with  at  least  50%  statistical  confidence  by  end  of  the  first  full-up, 
system-level  developmental  test  event  of  SDD 

-  Threshold  value  must  be  approved  as  a  part  of  the  TEMP,  and  recorded  in  the  SDD  contract 
and  APB  at  Milestone  B 

-  Requires  review  of  material  developer’s  reliability  case  documentation 

•  AMSAA  and  AEC  to  apply  Reliability  Scorecard 


ATEC  to  perform  threshold  assessment,  and  lead  IPR  in  event  of  a  breach: 

-  PEO/PM  develops  corrective  action  plan 

-  AEC  performs  assessment  of  PM’s  plan  and  projected  reliability 

-  AMSAA/AEC  estimates  ownership  cost  impacts 

-  TRADOC  assesses  utility  of  system  given  current  reliability  maturity  level 

-  ATEC  CG  provides  recommendation  to  ASA(ALT)  thru  Army  T&E  Executive,  with  PEO 
coordination  in  advance 


ASA(ALT)  policy  expands  the  Army’s  current  T&E  mission 


1 .  Per  CJCSI  31 70.01  F,  JROC  “Interest”  refers  to  programs  that  have  a  potentially  significant  impact  on  joint  warfighting. 


UNCLASSIFIED 


ASA  (AL&T)  — 

Army  RAM  Improvement  Initiatives 

(AAE  Memo,  4  Sep  08) 


Army  PM  Charters  to  explicitly  include  RAM  focus 

APB  to  include  an  increased  RAM  scope  and  hold  PEOs  &  PMs  accountable 

ASARC  (&  other  reviews)  to  be  modified  to  focus  on  RAM 

Reliability  expertise  &  POCs  within  ASA(ALT)  SOS  Engineering  Organization 

RAM  emphasis  in  future  capabilities  documents  &  acquisition  contracts 

Improve  RAM  training  provided  to  Army  acquisition  &  logistics  workforces 

Sponsor  RAM  workshops  &  conferences,  including  latest  RAM  improvement  initiatives 

Encourage  use  of  GEIA-STD-0009  (Reliability  Stnd  for  Design,  Devel.  &  Manufac.) 


Apply  Reliability  Scorecard  early  to  evaluate  progress  in  the  development  process 


UNCLASSIFIED 


ASA 

5-Step  Army  Policy  Implementation  Plan 


Technology  Development  Phase 


System  Development  &  Demonstration  Phase 


Establish 


Establish  test  threshold  value1-2 


Default  value  is  70%  of  requirement,  and  must  be 
demonstrated  with  at  least  50%  statistical  confidence. 


Document 


CDD6 


RFP 


1,2,4 


►  SDD  Contract1-2-' 


RIWG  sample  RFP  language  (DAU  Website),  GEIA-STD-0009 


TEMP1-2 


APB1 


Threshold  to  be  approved  as  part  of  TEMP  and 
incorporated  in  SDD  contract,  TEMP,  and  APB. 


Plan 


Develop  RG  Planning  Curve1-2 


Source  Selection  Support1-4 


AMSAA  Reliability  Growth  (RG)  Methodology 


Breach  Contingency  Planning1 -2-3-5 


Evaluate 


Evaluate  RG  Plan1-2-4 

Threshold  Assessment2 

Report 


ESR/CIPR2 

SEP2 

RIWG  Reliability  Engineering  Scorecard  (DAU  Website) 


Sys  Eng.  Plan1-2 


Milestone  B  OAR  Risk  Assessment2 


Identify  LCC  Impacts1-3-5] 


M 


Threshold  Breach  Report2’35  j 


I 


Only  done  if  threshold  breached.  I 


OTA  Assessment  Report2 


Key  players:  1  PEO/PM,  2  AEC-RAM,  3  AEC-ILS,  4  AMSAA  -  Reliability  Branch,  5  AMSAA  -  Resource  Studies  Branch,  and  6  TRADOC. 
Documentation:  Currently  developing  an  ATEC  guide  on  this  implementation  plan  and  associated  reliability  growth  planning  guesses. 
Reference:  ASA(ALT)  Memorandum,  Dated  6  December  2007,  Subject:  Reliability  of  U.S.  Army  Materiel  Systems. 

GEIA:  Government  Electronics  and  Information  Technology  Association. 


UNCLASSIFIED 


ASA  (AL&T) 


Summar 


The  Army  is  modernizing  &  transforming 


The  Army  must  organize  for  success 


SoS  Systems  Engineering  plays  a  pivotal  role 


UNCLASSIFIED 


ASA  (AL&T) 


Questions? 


14 


UNCLASSIFIED 


ASA  (AL&T) 


Back 
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UNCLASSIFIED 


ASA 

LWN/BC  Governance 


Tiers 


1 


Supporting  LWN/BC 
Capability  Set 
Development 


Management 


Execution 


Architecture 


CSA/VCSA 

Evolves  vision,  guidance  and  directions 
Identifies  issues  for  decision  making 
Makes  decisions 


CPM 


G3/5/7  LWN/BC  Lead 

•  Manages  decision  making  process 

•  Prioritize,  synchronize,  integrate 
Frames  and  coordinates  ad  hoc  and 
continuing  task  execution 


Leads  for  All  SoS  Process 

•  Executes  tasks 

Collaborates  among  SoS  Processes 


LandWarNet  Directorate,  G3/5/7 


4  February  2008 


America’s  Army:  Strength  of  the  Nation 
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UNCLASSIFIED 


ASA 

LWN/BC  &  SoS  SE  Synchronization 


Tier  1/2 


-►  Support 


17 


The  success  of  the  LandWarNet  strategy  is  reliant  on  the  transformation  of 
current  Generating  Force  processes,  policies  and  procedures. 

The  adoption  of  a  System  of  Systems  Engineering  Approach  is  the  first  critical 
step  in  the  transformation  process. 

Concurrently,  other  processes  must  adapt  to  enable  the  System  of  System 
approach.  The  Generating  Force  processes  identified  for  transformation 


in< 


Operational  Analysis  (M&S) 
Programming 
Testing  &  Certification 
Information  Assurance 
Fielding  Capability  Sets 
Acquisition 

*Prioritization  (DARPL/ARFORGEN) 


Architectures 

Configuration  Management 
Portfolio  Management 
Capabilities  /  Requirement  Validation 
Force  Integration  &  Documentation 
(TO&E,  BOIP) 


Q 


To  achieve  synchronization: 

Must  determine  critical  deliverables 
ID  organizational  Interdependence 
Target  key  decision  points  (strategic  and  operational) 
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UNCLASSIFIED 


-  ASA 

Overview  of  SoS  SE  Activities  -  FY09 


PRIORITIES 

FY08 

FY09 

4Q 

IQ 

2Q 

3Q 

4Q 

UBC 

UBC  120  Day  Study  Complete 
|  Initial  Systerr 

s  Views  for  APOM  11-15  (Capat 

ility  Sets  FY11-12  &  FY13-14) 

Design 


▲  Army 
Decision 

Update  En;erprise  Arch  ,  plan  for  POM  1 2-|1 7 

Process  Recommendations  frojn  UBC  120  Day  Study  Lessons  .earned 

Candidate  Systems  for  Capabili 


LWN/BC 
Capability  Set 
Development 


Army-USMC 
C2  SA 

Convergence 


y  Packages 
Capability  Set  COA  Evaluations 


Army 


Decision 


Baseline  Cap  Set  Arch 


Capab  ility  Set  BOI  and  Cost  Analysis 


Implementation  Plan 

APOM  11-1|5  Impact  Analysis 


Manage  Implementation 


C2  SA  Con/ergence  Architecture 


Wideband 

Interoperbility 

Study 


UAS  Interoperability  Task 
Review  of  CIO/G-6  AWIP 


Army 
1  Decision 


Assessment/selection  of  COA;  (HASC  CDL  Report 

Enterprise  Architecture  Update 


Tactical 

CDS 


Current  Implementations/Capapility  Needs 

Eval  Candidates  &  COA  Arch 


COA  Assessment  /Selection 


Data  Strategy 


Assess  Stakeholder  Positions 


Recommend  COAs 


TODAY 


Assess  Impact  on  PORs 


Review  Implementation 


TORs  Being  Developed  for  Each  Activity 
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Headquarters  U.S.  Air  Force 

Integrity  -  Service  -  Excellence 

USAF  Systems  Engineering 


Presentation 


to  the  NDIA  Systems  &  Software 
Engineering  Conference 

21  Oct  2008 


U.S.  AIRFORCE 


Mr.  Terry  Jaggers,  SES 
Deputy  Assistant  Secretary 
Science,  Technology,  and  Engineering 


Agenda 


U.S.AIR  FORCE 


■  AF  Systems  Engineering  (SE)  Revitalization 


■  AF  Early  SE  Defined 


■  AF  Early  SE  Initiatives 


■  Early  SE  Workforce  Considerations 


Integrity  -  Service  -  Excellence 


AF  SE  Revitalization 


U.S.AIR  FORCE _ 

■  Accomplishments 

>  Published  first  AF  Instruction  on  Systems  Engineering  (Jul  07) 

>  Approved  first-ever  software  development  guide 

>  Approved  AF  life  cycle  prototyping  policy 

>  Approved  SE  plan  (SEP)  policy  and  streamlined  staffing  by  80% 

>  Developed  integration  readiness  assessment  tool  and  implementing  during  AF 
ACAT  1C  program  support  reviews 

>  Funded  interface  management  program  for  CSB  support  (FY10  POM  initiative) 

>  Established  concept  development  SE  plan  (ConSEP)  and  concept  spec  (CCTD)  for 
space  pre-A  systems  engineering 

>  Established  AFIT  SE  Graduate  Program  and  SE  Masters’  Degree  Programs 

>  Co-sponsored  NRC  Study  to  define  25-yr  AF  STEM  requirements 

■  Initiatives 

>  Updating  AF  Scientist  &  Engineer  Strategic  Plan  (goal  3  focused  on  system  & 
software  /  specialty  engineers  for  pre  and  post-A  materiel  development) 

>  Developing  standardized  program  support  reviews  for  all  AF  programs 

>  Collapsing  discrete  S&T  and  engineering  polices  to  form  seamless  Research, 
Development,  &  Engineering  Policy 

>  Standardizing  pre-A  ConSEP  and  CCTD  policy  &  processes  across  the  AF 


Integrity  -  Service  -  Excellence 
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w 


U.S.AIR  FORCE 


AF  Supports  NRC  Early  SE 
Recommendations  ” 


A  A 


A 


Material 

Solutions 

Analysis 

Technology 

Development 

Engineering  & 

Manuf  actur  ing  D  eve  1  opment 
&  Demonstration 

Production  & 
Deployment 

Operations 
&  Support 

^  .  * 

i 

ix* 

System  Life  Cycle  Acquisition  Process 

i  i  i 

Combat  Dev 

i 

<3 - a 

UR 

/eloper 

1 

Materiel  Developer 

PM  Total  Life  Cycle  System  Manager  M< 

|  »  | 

- U- 

ateriel  Command 

- E> 

Acquisition  Framework 


High 

Ability  to 
Influence 
LCC 
(70-75% 
of  Cost 
Decisions 
Made) 


Less  Ability  to 
Influence  LCC 
(85%  of  Cost 
Decisions 
Made) 


Little  Ability  to 
Influence  LCC  (90-95% 
of  Cost  Decisions 
Made) 


(5% -10%) 


28%  Life  C j  rcle  Cost 


Minimum  Ability  to  Influence  LCC  (95%  of  Cost 
Decisions  Made) 


I 


72%  Life  Cycle  Cost 


Adapted  frorrt  "Pre-Miiestone  A  and  Eariy-Fhase  Systems  Engineering:  A  Retrospective  Review  and  Be  fie  fits  for  Future  Air  Force 
Systems  Acquisition,"  Air  Force  Studies  Board,  National  Research  Councii  of  the  Natiof^ai  Academies,  Washington  DC,  2008. 

Original  publication:  Andrews,  Richard,  "An  Overview  of  Acquisition  Logistics,"  Defense  Acquisition  University,  Fort  Befvoir,  VA,  2003. 


Integrity  -  Service  -  Excellence 
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AF  Early  SE  Defined 


U.S.AIR  FORCE 


Material 

Solutions  Analysis 

Technology 

Development 

Pre-AoA 

!  AoA  SoS 

Pre-Program 

Traditional  Design 

Concept(s)  SE 

Concept(s)  SE 

■  Requirements  SE 

Systems  SE 

•  “Big  A”  trades 

i 

i  •  Refine  mat’l  concepts 

i 

[•  System  requirements 
| 

•  Preliminary  design 

•  Use  SE  to 

i  •  Use  SE  to 

]•  Use  SE  to 

•  Use  SE  to 

-  Mature  requirements 

-  Inform  requirements 

i  -  Define  requirements 

-  Trace  requirements 

-  Determine  tech  needs 

i  -  Refine  tech  needs 

|  -  Develop  tech  roadmaps 

-  Assess  technology 

-  Develop  mat’l  concept 

-  Refine  concepts 

i  -  Describe  system 

i 

-  Describe  design 

•  Lab,  Product  Centers, 

i 

|  •  MAJCOM-Led  AoA 

i 

i*  Product  Center  XR  or 

•  Program  Office 

MAJCOMs  &  Industry 

i  Team 

i 

Program  Office 

i 

Concept  Spec  (CCTD)  - > - Spec - > 

-  ConSEP  - > -  SEP - > 


AF  Instruction  for  Life  Cycle  SE  (AFI  63-1201)  - > 


Integrity  -  Service  -  Excellence 
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AF  Early  SE  Initiatives 


U.S.AIR  FORCE 


■  Programs 

>  Increased  pre-program  engineering  &  analysis  by  39%  (+$37M/yr)  in  FY10  POM 

>  Increased  pre-program  prototyping  by  26%  (+$10M/yr)  in  FY10  POM 

■  Policy  &  Process 

>  Approved  CCTD  guide  and  directing  ConSEPs  /  CCTDs  for  all  pre-program  concept 
development  in  lab  and  product  centers 

>  Directed  prototypes  IAW  OSD  policy  (expect  to  see  in  FY11  or  12  POM) 

>  Multiple  AFS021  process  initiatives  (capability  planning  &  tech  assessment) 

■  People 

>  Established  AF  Technology  Transition  Office  to  oversee  BA-4  policy  &  programs 

>  Increased  pre-program  AF  civilian  engineers  at  MAJCOMs  by  5% 

>  Identifying  military  engineers  from  AF  military  plus  up  (31 6K  to  330K) 

>  Designating  level  III  SPRDE-PSE  Chief  Engineer  positions  in  pre-program  developmental 
planning  offices  at  product  centers,  in  addition  to  program  offices 

>  Designating  level  III  SPRDE-SE  Chief  Engineer  positions  in  AF  Research  Lab 

>  Updating  S&E  Strategic  Plan  to  address  early  SE  and  specialty  engineering  competencies 


Integrity  -  Service  -  Excellence 
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Early  SE  Workforce  Considerations 


U.S.AIR  FORCE 


■  The  numbers  of  “illities”  specialists  to  adequately  contribute  to  early  SE  (RAM, 
manufacturing,  ESOH,  HSI,  etc)  will  have  to  be  addressed 

■  Requirements  officers,  lab  technologists,  and  product  center  developers  should  all 
have  SE  training  (unlike  post-A  development,  pre-A  planning  is  a  team  sport) 

■  Offices  doing  early  SE  should  be  staffed  by  a  seasoned  &  experienced  workforce 

■  Early  SE  work  priorities  should  be  set  at  the  4-star  level  and  not  relegated  to  the 
early  SE  staff  to  guess  (in  this  phase,  everything  can  be  chased) 

■  Critical  acquisition  positions  (CAP)  should  be  considered  for  concept  developers 
outside  of  established  program  offices  (for  continuity  and  leadership) 

■  Writing,  reviewing  &  approving  early  SE  concept  specs  (CCTDs)  will  require  new  or 
adapted  skills  (or  at  least  a  revitalization  of  old  ones) 

■  Pre-program  prototyping  will  require  integration  risk  and  EMDD  skills 


Integrity  -  Service  -  Excellence 
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U.S.  AIR 


Integrity  -  Service  -  Excellence 
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Systems  Engineering  and  Capability 
Portfolio  Management  (jno  Approach) 

21  October  2008 

NDIA  Systems  Engineering  Conference 

fK\ 

People  throughout  the  trusted,  dependable  and  ubiquitous  network  are  empowered  by  their 
ability  to  access  information  and  recognized  for  the  inputs  they  provide. 


C3-NII 


Information  &  the  GIG  -  Layered  Perspective 


v  Defined  data  strategy  -  attributes  set  by  applications 
s  XML  driven  by  DoD  directives 
v  SOA  enterprise  environment  with  managed  services 


v  IP  based  with  QoS  established  by  applications 
v'  Multi-media  for  highly  available  communications 


nsport  o 


Nwk  Mgmt  are  critical  components 


s  Loosely  coupled  applications  based  upon  SOA/SLA 
v  Enabled  applications  are  highly  adaptive  and  flexible 


m 

I 


Assured  information  (data)  access  is  the  critical  concept 
-  the  user  sets  the  information  access  requirements 


Global  Information  Grid  (GIG)  Transport  Tiers 


Tier  7  -  Backbone  (DISN-NG,  WGS, 
Teleports,  Comm  SatCom,  JNN) 


'l-  Intermediate  (WIN-T,  JTRS,  W< 
AEHF,  NMT,  HC3,  FAB-T,  TSAT) 


Tier  3  -  Edge  (JTRS,  MUOS,  HC3,  NMT, 
TSAT) 


Major  commercial 
equipments 


Tactical  equipment 
mix 


Mostly  mUttery  with 
some  comnjdrciiei 
equipmentmx 

$ 


o  GIG  is  an  IF  unified  network  having  a  BLACK  routing  and  switching 
basis  -  tiered  in  many  respects  as  commercial  networks;  with  cost 
significantly  increasing  towards  the  edge 

OO-INII 


9 

Incomplete  Network  Solution  -  Losing  Sight  of  the  Network 


Tier  1  -  Backbone  (GIG-BE, 
TSAT,  Teleports) 


Tier  2  -  Intermediate 
(WIN-T,  JTRS,  WGS) 


Tier  3 


Edge  (JTRS,  MUOS) 


Network  Topology  Relationships 


Enterprise  servi 
repository  locat 


Distance  (topology 
metric)  is  critical 


Aggregated  date  flow  at 
the  node  is  a  critical 
attribute 


Security  boundaries 


Integrated  Network  Solution 


o  Understanding  the  entire  network  is  critical  so  to  not  compromise 
a  cost  and  warfighter  effective  solution  (Interoperability) 

o  Forcing  the  core  and  tactical  edge  networks  to  be  addressed  an 
Integrated  structure 

o  Network  and  Enterprise  programs  are  NOT  independent 

o  Network  is  part  of  the  GIG  -  requires  relationship  to  the  services 
and  applications,  BUT  information  (data)  is  the  critical  element 

o  Interoperability  with  more  than  a  single  Service  element  or  a  partial 
force  -  total  force  including  the  all  Services  and  coalition  forces 


GIG 


Conduct  portfolio  management  of  enabling  programs  and  capabilities-develop  material  and  non¬ 
material  solutions  to  ensure  timely,  synchronized,  and  integrated  delivery  of  Net-Centric 
capabilities 


User 


Applications 


I  ^  I 


C2 


BA 


BP 


Protect 


LOG 


FS 


CMS 


Net  Centric  Infrastructure 


Transport  g  Services  g  Transport 
Net  Management 


EHi_ 


A 


Provides  an  end-to-end  capability 


J 


User 


Applications 


I  ^  | 


C2 


BA 


BP 


Protect 


LOG 


FS 


CMS 


NC  portfolio  is  an  enabling  infrastructure  for  other  Capability  Portfolios 
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JNO  (NC)  CPM  Roles  &  Responsibilities 


Planning  &  Synchronization 


Enabling  Infrastructure 
Portfolio 


Resource  Mgmt 


Integrated 

Capability  Capability  Assessment 
Delivery 


Systems  Engineering 


Recomme  relations 


Capability  -  JCIDS 


Balance 


Acquisition 


•  JNO  CPM  is  executing  four  areas  of  responsibility 

•  Resource  Management  and  Capability  Assessment 
are  focus  for  PR/POM  actions 

«  System  tools  and  Acquisition  are  being  worked 


Use  JNO  portfolio  management  to  improve  synchronization,  interoperability  & 

integration  --  MJej/jcu  cost,  schedule,  &  performance  frisk))  across  the  portfolio 
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Capability  Increments 


JNO  Tier  2 

Strategy  c  Interest  Summary 

Warfi loiter  Needs 

Area 

(QDR,  S  PG,  J  FG  ) 

(J  NO  S  VttrF.  OGA,  IPL  i ,  JUONl,  NCOEJCD,  Op  i 
Plani ) 

Ndv*Drk 

Goals 

Manage  rr 

Knoviedj  *  Balance  portfolio  based  on  a  F 
h/feria^ri  ■  Provide  diility  to  collaborate  a 

Informal 

Transpor  ■  Provide  a  hiyhty  available  netv 

- ■  Provide  d)ility  to  identifystorr 

Erterpris 

s ericas  ■  Protect  interjity  of  data  and  s^ 


Capability  Increments 


'  Ensure  integrated  infrastmctui 
resource  (network,  sped  run,  s 
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o  Defines  Near-,  Mid-,  and  Far-Term  capability  deliveries 
o  Capability  Increments  will  be  approved  via  the  FCB  and  SWarF 


Network  Architecture  Perspectives 


airborne 


Terrestrial 


NWK  Layers 


,ntermediate . 


TdCtin^l 
-  ~Iactical 


Access  Layers 

(offering  network 
access  points) 

*  Understanding  the  network  framework  (architecture  topology)  is  essential  to 
determining  the  ability  to  meet  the  warfighter  capability  demands  and  optimizing 
the  investment 


o  The  space  and  airborne  access  layers  are  not  necessary  networks  but  offer  the 
networks  an  alternate  media  means  not  available  within  the  nwk  physical  domain 


NWK  Operating 
Environment 

e  Network  perspective 

o  Network  elements  inducting 
services  which  support  the 
Time  terrestrial  warfighter 

9  Depict  the  supporting  GIG 
Network  elements  required 
capabilities 


NC-lnv 


waveform:  HNW 

Rarte:  Max  22Mbps,  mean  1 1Mbps,  var  8  Mbps 
Dis\Max  7kM,  nom.  2  kM  ! 


Waveform:  HNW 

Rsite:  Max  22Mbps,  mean  1 1  Mbps,  var  8  Mbps 
Dii:  Max  7kM,  nom.  2  kM 


Waveform:  HNW 
Max  22MbpJs,  mean  1 1  Mbps,  var  8  Mbps 
I  Dis:  Max  7kM,  nom.  2  kM 


Satellite 


Satellite 


Satellite  Satellite 


Satellite 


Satellite- 


Satellite 


Waveform:  HNW 


Aggregated  traffic 


ES  clearly  identified 


link  avaiiauiiiLy. 

Trans  ERIP:  +28dBV\( 
Link  margin:  2. IdB  |j 
IA:  COMSEC  and  TRtf 


Waveform:  HNW 

Rate:  Max  22Mbps,  mean  1 1  Mbps,  var  8  Mbps 
Di$:  Max  7kM,  nom.  2  kM 


Waveform:  HNW  | 

Rate:  Max  22Mbp$,  mean  1 1  Mbps,  var  8  Mbps 
Dis:  Max  7kM,  norin.  2  kM 


Waveform:  HNW 
Rate:  Max  22Mbps, 
Dis:  Max  7kM,  nonrj 


,02kM 


lOOOkM 


Waveforms  clearly  identified 


The  Characteristics  of  a  Terrestrial  Tactical  Network 

Architecture 


Individual  links  with  attributes 


Hierarchical  nodes 


Waveform:  HNW 

Rate:  Max  22Mbps,  mean  1 1  Mbps,  var  8  Mbps 
Dis:  Max  7kM,  nom.  2  kM 


Peer  nodes 


Waveform:  HNW  I 
Rate:  Max  22Mbps,  mean  1 1 
Dis:  Max  7kM,  nom.  2  kM 
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LOS 


Terrestrial  NWK 

Network  is 
hierarchical  in 
structure 

Distance  is  critical 
with  hierarch  topology 

Diversity  is  key 

Space  is  an  extension 
of  terrestrial! 

Airborne  Nwk 
Distance  is  in  terms  of 
near  and  far 

Diversity  is  important 

C2  nodes  with  ES  are 
critical 

Position  in  air  space 
relative  to  permissive 
environment  is  key 


Alternate  links  (waveforms) 


Traffic  statistical  attributes 


Distance  based  nodal  separation 


NC 
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Assessment  and  Framework 


JNO  Capability  Increments  (2) 


Sj  “ssiZl 
i 


i  a  * 

_  - 

-  Pb  l i i  HU  Tinki 
.'in  >b  umiM 

-  Fahlld  ibl.JFH 

^ILBLI 

■  iXV  ii??: 

■  fTQ  i?  ir 

■  am  n?  ?  Ii 

■  ir-ui  id 

•  h  UlLII  |llfal 

•  rdaaiLmla.  h  ■ 

•  Hklbibl  Fl^bI' 

■  ■Ha  ih 

1 

3 

•  '.'lit him  dfai  i 

■  ■.L^ll 

•  Fafalld  FjIb  Ibk 

•  PlI  bail  la 

-  fa'UI  i a h  i  di-hlil 
b'i? r- 

-  .\a  -  ii ■  a ■  i a  ■  1 

hi  7  A  !■ 

- 

fa 

lilna  dn 

'■  rdilBllBBBb  Ul 

falllB  db  ■  |  B.lhBB 

■  ■.'■la  h  km  dl 
h  k  ■  dh  i  ■  ulb  a 

-kU.l.l 

■  i'faa.  fa  hfaia 

adli  ri.m  ■  lb.h 

■  i"  ta-1 

•  Fii  fad  lb  fa  a  1  ■  ■  d 

[  c. 

1 

L 

•  .'■■•■  lIiIiib  dfai 

lTL.Ti.h7 ' 

; 

I  alafah 

■  ilbiIIh  li-i 
lila.d  i  ■ 

•  Fill  id  .  Jak  rjc- 

;■  Fa  -ha  b  blF.'.i  i 
■  ■Hub 

7  :+h  m  ria  fa  hl?.\ 
ail.  -aafafa 

■.fa  ■  A'. 

'^u»hhi  ■FiIfilb.FW 

|  . . . 

'•  Jih  laha^L 

j  Fh  ?  .  .  -fa-  -1 

■  nuah.h.  i 

Fifa  |  1  fa  Lbl Ffa'3 

■  ■fari  ■aln  ■  ■ 

•  1  fa  hi.  i ■  Ha  ih 

■  F.Ubk  .  d-.  A. 

•  rniaiiLiii' 

'  iBihiyabuli 

-  LILI-I 1.  J.h  : 

Fa  I  a  di  faklihlad 

1  al  |  ia ha  ihi 

-  h  fahidK.  n.i 

-  Jifairi  ■■  l 

■  d-B  fa 

Art  ?  ■■BILB 

'  F  .Uik  1  hh'J fa' 

■  Hf.Hir 

•  IiIh  anfa 
liIU  ndh  i  A  VIP 

•  I'M  a  ■■n 
■  uiiy.idfai 

■  l'J  hll  Idhl  1 1 

J 


Capability  Delivery  increments  (CDi):  Describes 
desired  Capability  Delivery  over  time 
"  Derived  from  J ROC-approved  sources  (JIC,  JCD) 
■  Parsed  into  evolutionary  Increments  of  capability 


Quantified  CDi  w/Metrics: 
enables  the  technical 
analysis  of  the  portfolio 


^  Conti  red  Jci  rt  Task  Rirce  ( CJTF )  Modd  Army  HECT  Nd  wk  \vrth  IA/ES/NM 
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Architecture  Views:  Describe  the  POR 
capability  available 

■  Derived  from  JCIDS  documents 

■  Overlays  of  ES,  NM,  IA 

■  Assembled  into  evolutionary 
architectures  by  Increments 


C3-NII 
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Specific  Assessment  and  Analysis  Aspects 


14% 


□  Air  Force 
■  Army 

□  Navy 

□  USMC 

□  D^ense-v^de 


Portfolio  Financial  Profile:  boundary  and  baseline 
for  resource  analysis  and  optimization 

■  286  Program  Elements  (whole  or  partial) 

■  $160B  total  over  FY08-13  (RDT&E+Proc+O&M) 


Integrated  Master  Schedule:  analysis  of  program, 
interdependencies,  and  synchronization  issues 

"  Provides  support  to: 

-  Architecture  Development 

-  Program  and  cross  portfolio  analysis 
■  POM  focus  teams 


Ops  Impact  Analysis:  quantifies  impact  of  portfolio 
changes  on  ops  effectiveness 

■  OPTEMPO 


ZD  14 

Tiim^frarrn? 


2013 


Lethality 

Survivability 


C3-NII 


Wired  Infrastructure  > 
lOGbps,  >  1000  Users 


Quantitative 

capability 

increments 


Large  Fixed  SATCOM  -  35  - 
110  Mbps,  >100  Users 


Airborne  SATCOM/Wireless  LOS  - 


Wireless  LOS/Small  SATCOM  -  Networking  to 
tactical  edge,  2Mbps/64  kbps  for  dismounted  user, 
10,000s  of  users  -  1 .5  Mbps  COTM  vehicle 


External  Balance 


Balancing  the  Portfolio  -  Resource  Process 


Capability 

increments 


Strategic 
interests  & 
warfighter 
capabilities 


Program 

capability 

increments 


BA,  JC2,  other 
Portfolios 


Demand 


Balanced 

Portfolio 


Focus 


Goals 

■  Balance  portfolio  based  on  a  Prioritized  Capability  fulx 

■  Provide  irility  to  collaborate  and  sharesituationai  awareness 

■  Provide  a  hicfily  available  network  extending  to  the  first  tacticd  mile 

■  Provide  Utility  to  identifyst  ores  hare  exchange  data'irifonrction 

■  Protect  integrity  of  data  and  system; 


■  Ensure  integrated  infrastructure  situation^  awarenes  to  enable  efficient 
resource  (network,  street  run,  setvita 


Assess 

Quantified 

Capabilities 
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Application 

Demand 


NC 
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Relative  Improvement  (%) 


Analysis  Example 

#  of  BDE  SATCOM  terminals  required  to  connect  the  edge 


Technical 


Time  to  Achieve  Objective 


1.  Define/Model  operational  vignettes 

2.  Assess  performance;  connectivity,  availability 
Message  Completion  Rate  (MCR) 

3.  Assesses  OPTEMPO,  Lethality,  Survivability 


/  — 

2008  Baselin 

•  H 

•  3-BDE:  EPLRS,  SINCGARS 

•  1-RCT:  EPLRS,  SINCGARS 

•  DSCS,  UFO,  Milstar,  Comm 

<"»  O 

Based  on  MCO-3  Phase  lllb  combined  amphibious/ground  assault — 
designed  to  relieve  stress  on  broader  campaign 

Additional  analysis  for  other  MCO  scenarios  and  impact  of  cyber/space 
attack  planned  for  2007— per  DSD  draft  guidance 


JNO  recommendations  increase  Warfighting  effectiveness 


peratiorial 


,  Remaining  SATCOM 


Connectivity,  Message  Completion  Rate,  OPTEMPO,  Lethality  and  Survivability 


OPTEMPO  Improvement 


Ops  Impact  Analysis  Results 

Benign  Threat  Environment 


t 


4- 


Preliminary 


l 


2008 

2014 

Timeframe 

2018 


o  Network  equipped  forces  have  significantly  improved  OPTEMPO 
o  Good  Situational  Awareness  (SA)  &  Battle  Command  (BC)  result  in  predictable  outcome 


MCO:  Major  Combat  Operation 

NEO:  Non-Combat  Evacuation  Operation 
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Network  Performance  Analysis 


S  AT  COM  Subnet  In  Green 


Green  -  SATCOM 


White  -WNW 
global 

Purple  -  HNW 
All  Others  -SRW 


(CERDEC  Tool) 


White  -  WNW 
global 

Purple  -  HNW 
All  Others  -SRW 


o  Allows  quick  turnaround  studies  with  numerous  excursions  feasible 
o  Provides  Message  Completion  Rates  and  other  Network  characteristics 

9  Used  as  feeder  to  higher  fidelity  models  (e.g.,  OPNET)  and  provides  means  of  visualizing  / 
analyzing  high  fidelity  models 
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Expanded  View  of  JTRS-WIN-T  Connectivity  by  Waveform  with  Aerial  Layer 
Applied  and  FCS  Spinout  Items  From  Soldier  to  Brigade  Main 


Summary 


o  NC  Portfolio  balances  the  three  building  processes  -  capabilities, 
acquisition  and  resources 

o  NC  Portfolio  employs  a  Systems  Engineering  based  portfolio 
management  approach 

e  Achieves  a  quantitative  analytical  position  based  upon  warfigther  based 
capability  increments 

o  Places  the  capabilities  into  a  network  topology  architectural  framework 
which  is  used  to  offer  contextual  structure  to  the  capability  implementations 

e  Quantitative  demand  and  supply  concepts  are  used  to  evaluate  the  gaps  and 
overlaps  in  capabilities 

e  Implementation  /  program  solutions  developed  from  the  evaluation 
assessment  are  used  to  determine  the  right  investments 

9  Continual  analytical  assessments  for  the  three  building  processes  is 
done  using  a  combination  of  network  topology  architectures,  QCDI,  and 
modeling  tools 

o  Network  and  enterprise  services  performance  are  evaluated  quantitatively 

o  Specific  metrics  include  OPTEMPO,  lethality  and  survivability  derived  from 
operational  models  /  scenarios 
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High  Level  Topology  View 


Architectural 
structure  sets  the 
assessment  and 
analysis  context 

The  network 
topology  offers 
analysis  of  the 
links,  ES  and  IA 
aspects 


Operational  distances  will  likely  be  much  lower  in  heavily  foliated  or  urban  environments 

Loading  represents  the  traffic  generated  by  each  transmitter  (e.g.,  total  load  foraPLT  of  44  members  would  be  3.4  kbps  x  44  =  150  kbps ) 
Does  not  include  direct,  raw  ISR  feeds 


Terrestrial  Network 

o  Network  is  hierarchical!  in  structure 

o  Distance  is  critical  with  hierarch  topology  (node-to-node  -  peer-to-peer) 

9  Link  diversity  is  critical 

9  Space  and  UAVs  are  an  extension  of  terrestrial  -  these  are  access  points  (or  layers) 

9  Significant  work  is  need  to  insure  the  right  balance  exists  between  LOS,  space  and  UAV 

9  Throughout  the  implementation  consideration:  performance,  cost,  schedule  and  risk  need 
careful  assessment 
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JNO  Increments  to  Focus  Team  Solutions 


Joint  /  combined 
network 


lldier / 
litions 


Environment 


Fully  support 
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Quantitative 

Capabilities 


Enterprise  service 
repository  location' 


Enterprise  service 
repository  location' 


Aggregated  date  flow  at 
the  node  is  a  critical 
i  attribute 


Distance  (topology 
metric)  is  critical 


Distance  (topology 
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Network 

Architectui 


Rate  and  waveform 
is  a  denoted  metric 


GDF,  PDM,  Directed 


Focus  Teams 
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Gaps  /  Over  Laps 


Near  Term 


Far  Term 
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Integrated 

Schedule 
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Fight  via  bridge  to 
Joint  network 


Fight  Service  with 
Joint  Plugs 


Transport  to  1st 
Tactical  Mile  user 


Increase  legacy  g] 
intefdperability  m 


ring  limited 


Manpo 


connectivity 


■  Data  to  Brigade, 
MEU,  Ship,  Wing 


■  Best  Effort  Routing; 
bridged  voice 

•  Svc  command  DB 

■  Push  stovepipe 
sensor  data 

■  De-conflict  Service 
CERT  org  stds 

•  Joint  data 
protection 


Primarily  BLOS, 
secondarily  LOS 
Data  to  Bn/Co  level 
Initial  CES 
Metadata  &  tagging 
Federated  Portals 
Enforce  DB  standards 
Isolate  compromised 
Joint  CERT  standards 
Role-based  access 
Robust  CND;  detection 
&  response  to  attacks 


■  Support  dynamic 
task  reorganization 

•  Dismounted 
capability 

■  SOA  environment 

■  Enhanced  COI 
formation 

■  Machine  to  machine 
info  sharing 

■  CDS  (TS-S) 

■  Protected  &  assured 
in-transit 

■  identity  management 

■  COI  Security - 


operations 
Provide  comms  link 
services  (non-NW) 

Full  integration  of  all 
CES  (joint/service) 
Service  coreography 
to  dynamic  tasks 

Smart  push/pull 
Multi-national  MLS 
CDS  (S  -U) 

Assured  sharing 


■  Guaranteed  delivery 
of  data 

■  No  longer  platform 
centric 

■  Global  data  access 

■  Enhanced  CES 

■  Automatically 
integrate/fuse  multi- 
source  data 

■  ES  to  Tactical  edge 


■  Hardened  edge  to- 
edge  encryption 

■  Assured  info 
dissemination 


Simplified  Traffic  Network  Model  Structure 

(Far-Term  HIC  BDE) 


■  ■  -  ■  • 


A  DB  Viewer  ■  [Whiteboard  Session  in  Baseline  2.noi] 


File  Edit  Insert  Data  View  Hr 


All  |  Results  |  Directories  Services  | 


X 


Far-Term  HIC  BDE.mox 


£  .event 


11/23/2004 
Tue  12:00  Ad 


Import  DB  Text  File... 


MESA 

lbfc.de!  flfcjf 

tyybnager 


Server 

Type 

Manager 


Futanage  Ivtanage  I 

B yt esSent  F: c v d  Average  Latenc y  tv 
index  Index 

Routs  Routs 


Cols 


Cols 


Date:  5/1 3/2008 


Seivice 

Skp 

Equation 

F  - 

0  Value = RandomCalculate(1  G.  30. 1 800, 0)  /  60;  Used  Se 

0  Value = CurrentTime:  User2  St 

0  Value =1500 '1.2;  Uset3  Mi 

0  Value =56000:  U$er4  Oil 

0  Value = [Usei3 "  8|  /  Used  /  60:  //UserError("Ri  Uset5  =' 

0  Value = User3;  Parameter 

ojvalue = User3;  Parameter 

0  Value =Usei5;  Parameter  JH5? 

_0  il [CunentTime > [Used ♦ Usei2|l Value = 1 0; els  Parameter  Ani»»M  j  MB 


kllwMltral 


-  il 


liH.<r>  lft< 


l 


* 


I  It.-Jtt!  do  M  0 


SINCGARS  EPLRS 


SINCGARS  EPLRS 


SINCGARS  EPLRS 


SINCGARS  EPLRS 


© 


il L 


C3-NII 


ES  Network  Location  -Throughput  and  Cost 


Configuration  1 :  Total  throughput  achievable  versus  message  completion  rate  for  the 
mid-term,  high-intensity  conflict  configuration  with  Enterprise  Servers  at  the  BDE  level 
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Configuration  2:  Total  throughput  achievable  as  a  function  of  offered  traffic  load  for  the 
far-term,  high-intensity  conflict  configuration  with  Enterprise  Servers  at  the  CO  level 


o  The  location  of  the  ES  may  have  potentially  significant  effects  on  the  network  performance 
and  more  importantly  on  the  effective  network  throughput 

»  The  balance  of  ES  cost  vs  the  lower  level  network  cost  is  an  important  aspect  which  is  being 
currently  assessed 
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Operational  View  -  OV-1  (U) 


Team 


High  altitude-connectivity  to  the  ground  and  other  Joint  users 
Medium  altitude  are  responsive  to  the  commander 
Low  altitude-connectivity  to  the  soldier  in  urban  terrain, 
when  all  other  means  of  connectivity  may  be  lost. 
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UNCLASSIFIED  (U) 


Without  Satellite  Connectivity 


Slant  View 


Standard  View 


White  -  WNW  global 
All  Others  -SRW 


White  -  WNW  global 
All  Others  -SRW 


Network  Topology 


•  SRW  subnets  are  tightly 
clumped,  good  connectivity. 

•  Global  WNW  subnet 
connects  some  SRW 
subnets  but  is  fragmented. 

•  SATCOIUI  terminal  in  each 
SRW  subnet  connects  those 
without  Global  WNW 
connectivity. 


TNGFBCT 


SATC'OM  Subnet  in  Green) 


TNGFBCT 


SATC’OM  Excluded) 


White  -  WNW  global 
All  Others  -SRW 


Range  and  Capacity  Analysis 


Waveform/Radio  Range  Performance 


SINCGARS  Dismounts:  Tirem  +  Foliage 
SINCGARS  Vehicles:  Tirem  +  Foliage 
SRW  Vehicles:  Tirem  +  Foliage 
SRW  Dismounts:  Tirem  +  Foliage 
EPLRS:  Tirem  +  Foliage 
WNW  Local:  Tirem  +  Foliage 
WNW  Global:  Tirem  +  Foliage 
WNW  Global  w/UAV:  Tirem  +  Foliage 
HNW  Ground:  Tirem  +  Foliage 
HNW  w/UAV:  Tirem  +  Foliage 


8  10  12 
Range  (km) 


o  Compute  link  closer  and  capacity  for  given  network  laydown,  terrain,  and  vegetation 
o  Waveform  performance  analysis  feeds  Network  performance  analysis 


NOTES 

Masts  are  7  meters  for  HNW 

UAVs  at  low  altitude  (FCS  CL  IV  altitude  used  in  PM  FCS  BCT  analyses) 
Assume  80-90%  confidence 
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Representative  Results 
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v 


Number 

Effective 

And 

Suitable 


Cumulative  IOT&E  Results  Through 

FY  2008 


DSB  DT&E  Taskforce  Main  Conclusion 

May  2008 


•  “ .  .  .  the  single  most  important  step  necessary  to 
correct  high  suitability  failure  rates  is  to  ensure 
programs  are  formulated  to  execute  a  viable 
systems  engineering  strategy  from  the 
beginning,  including  a  robust  reliability, 
availability,  and  maintainability  (RAM)  program, 
as  an  integral  part  of  design  and  development. 
No  amount  of  testing  will  compensate  for 
deficiencies  in  RAM  program  formulation.” 
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Section  231  Report  to  Congress 
Core  T&E  Principles 


1.  T&E  should  concentrate  on  measuring  improvements  to  mission  capability  and 
operational  support  based  on  user  needs; 

2.  T&E  programs  should  experiment ....  learn  and  understand  the  strengths  and 
weaknesses  of  a  system  and  its  components,  and  the  effect  on  operational 
capabilities  and  limitations; 

3.  DT  and  OT  activities  should  be  integrated; 

4.  T&E  should  begin  early,  be  more  operationally  realistic,  and  continue  through  the 
entire  system  life-cycle; 

5.  Evaluation  should  be  conducted  in  the  mission  context  expected  at  time  of  fielding  to 
the  user ...  in  terms  of  operational  significance; 

6.  Evaluations  should  include  a  comparison  against  current  mission  capabilities; 

7.  Evaluations  should  take  into  account  all  available  data  and  information; 

8.  T&E  should  exploit  the  benefits  of  appropriate  M&S. 


New  Acquisition/T&E  Policies 

McQueary/Finley  Memo  on  Assessment  of  Op  Test  Readiness 

(21  May  2007) 


•  The  DUSD(A&T)  shall  conduct  an  independent 
Assessment  of  Operational  Test  Readiness  (AOTR)  for 
all  ACAT  ID  programs  and  special  interest  programs 
designated  by  the  USD(AT&L) 

•  The  CAE  shall  consider  the  results  of  the  AOTR  prior  to 
making  a  determination  of  materiel  system  readiness  for 
IOT&E. 


New  Acquisition/T&E  Policies 

Young  Memo  on  Competitive  Prototyping  (19  Sep  2007) 


•  All  acquisition  strategies  requiring  USD  (AT&L)  approval 
must  be  formulated  to  include  competitive,  technically 
mature  prototyping  through  MS  B. 
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New  Acquisition/T&E  Policies 

Young-McQueary  T&E  Policy  Letter  -  (22  Dec  2007) 


•  DT  and  OT  test  activities  shall  be  integrated  and  seamless 

•  Evaluations  shall  include  a  comparison  with  current  mission  capabilities 

•  T&E  should  assess  improvements  to  mission  capability  and  operational 
support  based  on  user  needs 

•  To  more  effectively  integrate  DT  and  OT,  evaluations  shall  take  into  account 
all  available  and  relevant  data  and  information,  including  contractor  data 

•  Operational  evaluators  will  continue  to  fulfill  their  statutory  roles  in  providing 
assessments  of  operational  effectiveness,  operational  suitability,  and 
survivability  to  the  Milestone  Decision  Authority 

•  To  realize  the  benefits  of  modeling  and  simulation,  T&E  will  be  conducted  in 
a  continuum  of  live,  virtual,  and  constructive  environments. 


New  Acquisition/T&E  Policies 

McQueary-Finley  Memo  on  Reliability  Improvement  WG 

(15  Feb  2008) 


•  Ensure  programs  are  formulated  to  execute  a  viable 
systems  engineering  strategy,  including  a  RAM  growth 
program. 

•  Ensure  government  organizations  reconstitute  a  cadre  of 
experienced  T&E  and  RAM  personnel. 

•  Implement  mandated  integrated  DT  and  OT,  including 
the  sharing  and  access  to  all  appropriate  contractor  and 
government  data  and  the  use  of  operationally 
representative  environments  in  early  testing. 


New  Acquisition/T&E  Policies 

McQueary-Finley  Memo  defining  Integrated  Testing  (May  2008) 


•  “Integrated  testing  is  the  collaborative 
planning  and  collaborative  execution  of 
test  phases  and  events  to  provide  shared 
data  in  support  of  independent  analysis, 
and  evaluation.” 


New  Acquisition/T&E  Policies 

Young  Memo  on  RAM  Policy  (July  2008) 


•  The  Service  Secretaries  are  directed  to  establish  Service 
policy  to  do  the  following: 

-  Effective  collaboration  between  the  requirements  and  acquisition 
communities 

-  Development  contracts  and  acquisition  plans  must  evaluate 
RAM  during  system  design. 

-  Evaluate  the  maturation  of  RAM  through  each  phase  of  the 
acquisition  life  cycle. 


Senior  Leadership  Buy-In  of 
New  Reliability/T&E  Policies 


“Having  performance  is  important,  but  not  as 
important  in  most  cases,  as  having 
reliability.” 

-  Hon.  Donald  Winters,  Secretary  of  the  Navy  (Sept  3,  2008) 


Number  of  Failures  in  the  Field 


Initiatives  to  Improve  Reliability, 
Maintainability,  and  Availability 


Ownership  Cost 


Ln  (Improvement  in  MTBx) 


Phase  I:  Empirical  Research 

Reliability  Improvement  vs.  Investment 
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Improvement  Ratio 


Phase  IIA  (Basic  Model) 


Investment/APUC 
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Phase  III:  Notional  Example 

Effect  of  Reliability  Investment  on  System  Cost  (UAV) 


Update  on  Revisions  to  MIL-STD-882 


NDIA  11th  Annual  Systems  Engineering  Conference 
System  Safety  -  ESOH  &  HSI  Session  3C4 

San  Diego,  CA 

Robert  E.  Smith,  CSP 


October  22,  2008 
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Introduction 


MIL-STD-882  is  DoD’s  standard  practice  for  system  safety 

Considered  the  system  safety  “bible”  for  DoD  Acquisition 
Programs 

Identifies  system  safety  practices  for  both  the  program 
manager  and  contractor 

In  existence  since  1969  and  has  been  revised  several  times 
Last  revision  (MIL-STD-882D)  occurred  Feb  2000 


MIL-STD-882  History1 


•  MIL-STD-882- July  1969 

-  First  DoD  system  safety  standard 

-  System  safety  program  became  mandatory  on  all  DoD-procured  products  and 
systems 

•  MIL-STD-882A- June  1977 

-  Centered  on  the  concept  of  risk  acceptance  as  a  criterion  for  system  safety 
programs 

-  Required  introduction  of  hazard  probability  and  established  categories  for 
frequency  of  occurrence  to  accommodate  the  long-standing  hazard  severity 
categories 

•  MIL-STD-882B  -  30  March  1984 

-  Continued  evolution  of  detailed  guidance  in  both  engineering  and  management 
requirement 

-  More  emphasis  on  facilities  and  off-the-shelf  acquisition  was  added,  and 
software  was  addressed  in  some  detail  for  the  first  time 


1  Clifton  Ericson  II,  A  Short  History  of  System  Safety,  Journal  of  System  Safety,  May-June  2006. 
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MIL-STD-882  History1  (cont) 


MIL-STD-882B,  Notice  1  - 1  July  1987 

-  Expanded  software  tasks  and  the  scope  of  the  treatment  of  software  by  system 
safety 

MIL-STD-882C  -  19  Jan  1993 

-  Integrated  the  hazard  and  software  system  safety  efforts 

-  Individual  software  tasks  were  removed 

-  Safety  analysis  would  include  identifying  the  hardware  and  software  tasks 
together  in  a  system 

MIL-STD-882C,  Notice  1  - 19  Jan  1996 

-  Corrected  some  errors  and  revised  the  Data  Item  Descriptions 

MIL-STD-882D  -  10  Feb  2000 

-  Under  the  Military  Specifications  and  Standards  Report  (MSSR)  initiative,  MIL- 
STD-882D  was  considered  important  to  continue,  as  long  as  it  was  converted  to 
a  performance-based  standard  practice  -  what  you  want  vs.  how  to  do  it 

-  Task  descriptions  removed 


Average  time  between  revisions:  ~  8  yrs 


Purpose  of  Revision 


Initial  drivers: 

-  Government  and  Industry  wanted  to  bring  back  the  Task  Descriptions  from  MIL- 
STD-882C  to  make  them  readily  available  for  call  out  in  contract  documents 

-  Align  with  current  OSD  Acquisition  Systems  Engineering  policy  changes 

Subsequent  drivers: 

-  Adjust  the  organizational  arrangement  of  information  to  clarify  the  basic 
elements  of  a  system  safety  program  and  the  process  flow  among  them 

-  New  tasks 

-  Support  DoD  strategic  plans  and  goals 


Highlight  of  Changes 


Update  will  be  referred  to  as  MIL-STD-882D,  Revision  1 

Subtitle  added  to  emphasize  ESOH  integration  into  Systems 
Engineering 

-  “ESOH  Risk  Management  Methodology  for  Systems  Engineering” 

Standardized  definitions 

Rewrote  task  descriptions  to  clarify  and  dissociate  from  each 
other 

-  100-series  tasks  -  program  management  and  control 

-  200-series  tasks  -  design  and  integration 

-  300-series  tasks  -  design 

-  400-series  tasks  -  compliance  and  verification 

Emphasized  the  identification  and  derivation  of  applicable 
ESOH  technical  requirements 

Added  Hazardous  Material  Management  Process  (HMMP)  task 


Highlight  of  Changes  (cont) 


Matrix  description  updated 

-  For  severity,  dollar  value  on  losses  increased  for  today’s  program  dollars  and 
logarithmic  progression  applied 

-  For  probability,  finite  period  of  time  or  cycles  added;  “Eliminated”  level  added 

-  Matrix  rearranged  to  have  ascending  severity  on  x-axis 

»  Mishap  risk  assessment  values  and  categories  unchanged,  but  graphically 
looks  different  than  current  matrix 

More  emphasis  on: 

-  Establishing  a  collaborative  ESOH  effort  using  the  system  safety  process 

-  Providing  coordinated  ESOH  input  to  systems  engineering  to  maximize 
performance  by  minimizing  the  environmental  “footprint”  of  the  system  and 
improving  safety  of  personnel  and  the  system  itself 

“Appendix  A  -  Guidance  for  Implementation  of  an  ESOH  Effort 
has  been  updated 

-  Additional  detail  on  hazard  definitions  and  assessing  top  level  mishaps 

-  Software  safety  techniques  and  principles  reintroduced 


Coordination  Process 


DoD  ACQ  ESOH  IPT 

-  882  Working  Group  complete  IPT  recommended  draft 

-  Review  and  comments 

-  Resolution  of  comments 

-  Provide  the  IPTs  recommended  Draft  to  SAF/AQRE 

NDIASE  Division 

-  Review  and  comments 

-  Resolution  of  comments 

Formal  DoD  Coordination 

-  Standardization  community 


Current  Estimated  Completion  Date:  Mid  2009 


Conclusion 


•  Clarifies  terminology,  incorporates  current  policy  and  defines 
task  descriptions  to  improve  system  safety  practices 

•  Strengthens  integration  across  Environment,  Safety,  and 
Occupational  Health  and  into  Systems  Engineering  during  the 
acquisition  process 

•  Improves  consistency  of  system  safety  practices  between 
programs 


Booz  Allen  Hamilton 
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Questions? 


Robert  E.  Smith,  CSP 
Booz  Allen  Hamilton 
1550  Crystal  Drive,  Suite  1550 
Arlington,  VA  22202-4158 
703-412-7661 
smith_bob@bah.com 


MIL-STD-882D,  Rev  1  -  Severity  Categories 


TABLE  1.  Severity  Categories 


Severity 

Category 

Severity  Level 

Environment,  Safety,  and  Occupational  Health 

Mishap  Result  Criteria 

Catastrophic 

I 

Could  result  in  death,  permanent  total  disability,  loss  exceeding 
$10M.  or  irreversible  siani  Scant  environmental  impact. 

Critical 

n 

Could  result  m  permanent  partial  disability,  injunes  o:  occupational 
illness  that  may  result  in  hospitalization  of  at  least  three  personnel 
loss  exceeding  $1M  but  less  than  $10M,  or  reversible  significant 
environmental  impact. 

Marginal 

m 

Could  result  in  injury  or  occupational  illness  resultEng  in  10  or  more 
lost  work  days,  loss  exceeding  SI  DDK  but  less  than  $1M,  or 
reversible  moderate  environmental  impact. 

Negligible 

IV 

Could  result  in  injury  or  illness  resulting  in  less  than  10  lost  work 
davs.  loss  less  than  $100K.  or  minimal  environmental  impact. 

Dollar  value  on  losses  changed: 

•  Increased  for  today’s  program  dollars 

•  Logarithmic  progression  applied 


►  Current  MIL-STD-882D  Definition 


MIL-STD-882D,  Rev  1  -  Probability  Levels 


TABLE  2.  Probability  Levels 


Probability 

Name 

Probability 

Level 

Description* 

Frequent 

A 

Likely  to  be  experienced  several  times  by  a  system  within  a  12 
month  period:  a  probability  of  occurrence  greater  than  1Q']  over  12 
months. 

Probable 

B 

Likely  to  be  experienced  by  a  system  within  a  12  month  period;  a 
probability  of  occurence  less  than  10'"  but  greater  than  10""  over  12 
months. 

Occasional 

C 

May  be  experienced  by  a  system  within  all  month  penod:  a 
probability  of  occurrence  less  dim  ID"’  but  greater  dim  10" J  over  12 
moodis. 

Remote 

D 

Unlikely,  but  possible  to  be  experienced  by  a  system  within  a  12 
mondi  penod:  a  probability  of  occurrence  less  dian  10^  but  greater 
than  10'*  over  12  months. 

Improbable 

E 

Sc  unlikely,  it  can  be  assumed  occurrence  may  not  be  experienced  by 
a  system  within  a  12  month  period:  a  probability  of  occurrence  of 
less  than  1C'4  over  12  months. 

Eliminated 

F 

Incapable  of  occurrence.  This  category  is  used  when  potential 
hazards  are  iden rifled  and  later  eliminated. 

-  Finite  period  of  time  or  cycles  added  to  description 

-  “Eliminated”  level  added 


►  Current  MIL-STD-882D  Definition 


MIL-STD-882D,  Rev  1  -  Risk  Matrix 


TABLE  3.  ESOH  Risk  Assessment  Values 


Frequent  (A) 


Probable  (B) 

16 

Occasional  (C) 

18 

Remote  (D) 

19 

Improbable  (E) 

20 

Severity 


Negligible 

IV 

13 


Marginal 


9 

11 


14 

17 


Eliminated 


TABLE  4.  Risk  Categories 


Risk  Assessment  Value 

Risk  Category 

Risk  Acceptance  Level 

1-5 

High 

6-9 

Serious 

10-17 

Medium 

In  accordance  with  DoD  policy 

10-20 

Low 

21 

N/A  (eliminated) 

-  Matrix  rearranged  to  have  ascending  severity  on  x-axis 

-  Risk  assessment  values  and  categories  unchanged 


►  |  Current  MIL-STD-882D  Definition 


Backups 
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MIL-STD-882  Eight  Mandatory  System  Safety  Steps 


1 .  Document  the  system  safety  approach 

2.  Identify  ESOH  hazards 

3.  Assess  the  risk 

4.  Identify  risk  mitigation  measures 

5.  Reduce  risk  to  an  acceptable  level 

6.  Verify  risk  reduction 

7.  Review  hazards  and  accept  risk  by  appropriate  authority 

8.  Track  ESOH  hazards,  their  resolution,  and  residual  risk 
throughout  the  system  lifecycle 


Booz  Allen  Hamilton 
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Current  MIL-STD-882D  Severity  Definitions 


TABLE  A -I.  Suggested  mishap  severity  categories. 


Description 

Category 

Environmental.  Safety,  and  Health  Result  Criteria 

Catastrophic 

I 

Could  result  in  death,  permanent  total  disability,  loss 
exceeding  $  1M.  or  irreversible  severe  environmental 
damage  that  violates  law  or  regulation. 

Critical 

II 

Could  result  in  permanent  partial  disability,  injuries 
or  occupational  illness  that  may  result  in 
hospitalization  of  at  least  three  personnel,  loss 
exceeding  $200K  but  less  than  $1M.  or  reversible 
environmental  damage  causing  a  violation  of  law  or 
regulation. 

Marginal 

III 

Could  result  in  injury  or  occupational  illness 
resulting  in  one  or  more  lost  work  days(s),  loss 
exceeding  $10K  but  less  than  $200K„  or  mitigatible 
environmental  damage  without  violation  of  law  or 
regulation  where  restoration  activities  can  he 
accomplished. 

Negligible 

IV 

Could  result  in  injury  or  illness  not  resulting  in  a  lost 
work  day,  loss  exceeding  $2K  but  less  than  $10K.  or 
minimal  environmental  damage  not  violating  law  or 
regulation. 

Current  MIL-STD-882D  Probability  Definitions 


TABLE  A-II.  Suggested  mishap  probability  levels. 


Description* 

Level 

Specific  Individual  Item 

Fleet  or  Inventory** 

Frequent 

A 

Likely  to  occur  often  in  the 
life  of  an  item,  with  a 
probability  of  occurrence 
greater  than  10'-  m  that  life. 

Continuously 

experienced. 

Probable 

B 

Will  occur  several  times  in  the 
life  of  an  item,  with  a 
probability  of  occurrence  less 
than  1G'1  but  greater  than  W~ 
in  that  life. 

Will  occur  frequently. 

Occasional 

C 

Likely  to  occur  some  time  in 
the  life  of  an  item,  with  a 
probability  of  occurrence  less 
than  10°  but  greater  than  10'J 
in  that  life. 

Will  occur  several 
times. 

Remote 

D 

Unlikely  but  possible  to  occur 
m  the  life  of  an  item,  with  a 
probability  of  occurrence  less 
than  10'J  but  greater  than  10'“ 
m  that  life. 

Unlikely,  but  can 
reasonably  be 
expected  to  occur. 

Improbable 

E 

So  unlikely,  it  can  be  assumed 
occurrence  may  not  be 
experienced,  with  a 
probability  of  occurrence  less 
than  10'*  m  that  life. 

Unlikely  to  occur,  but 
possible. 

^Definitions  of  descriptive  words  may  have  to  be  modified  based  on  quantity  of  items 
involved. 

**The  expected  size  of  the  fleet  or  inventory  should  be  defined  prior  to  accomplishing  Em 
assessment  of  the  system. 


Current  MIL-STD-882D  Risk  Assessment  Matrix 


TABLE  A-m.  Example  mishap  risk  assessment  values. 


SEVERITY 

PROBABILITY 

Cacastroplnc 

Critical 

Marginal 

Negligible 

Frequent 

1 

3 

-| 

,r 

13 

Probable 

“1 

5 

9 

16 

Occasional 

4 

6 

11 

IS 

Remote 

S 

10 

14 

19 

Improbable 

12 

15 

17 

20 

TABLE  A-IV.  Example  mishap  risk  categories  and  mishap  risk  acceptance  levels. 


Mishap  Risk 
Assessment  Value 

Mishap  Risk  Category 

Mishap  Risk  Acceptance 

Level 

1  -  5 

High 

C  ompcnent  Acquisition 
Executive 

6-9 

Serious 

Program  Executive  Officer 

10-17 

Medium 

Program  Manager 

IB  -  20 

Low 

As  directed 

*  Representative  mishap  risk  acceptance  levels  are  shown  in  the  above  table.  Mishap  risk 
acceptance  is  discussed  in  paragraph  A .4 .4.7.  The  using  organization  must  be  consulted  by  the 
corresponding  levels  of  program  management  prior  to  mishap  risk  acceptance. 


Joint  Rapid  Scenario  Generation 

*  (JRSGjf^m^ 

Systems  Engineering 
October  2008  , 


Mr.  Ralph  O’Connell 
US  Joint  Forces  Command 
Joint  Capability  Development  (J8) 
Senior  Systems  Engineer 


JRSG  Problem  Statement 


Generation  of  scenario  data  sets  do  not  support  operational 
requirements  for  near  real  time  mission  rehearsal,  course  of 
action  analysis,  and  adaptive  planning. 


The  increasing  use  of  complex 
M&S  applications  requires  data 
with  greater  fidelity  with  a  rapid, 
production  time. 

There  are  common  capability 
gaps  that  transcend  all  domains> 

Combined,  Joint,  Services,  and 
Agencies  (C/J/S/A)  are 
developing  independent 
improvements  to  their  scenario 
generation  capabilities. 


JRSG  Activity  Model  &  Domain  Support 


No  one  in  is  responsible  for  orchestrating  the  DoD  enterprise  solution 


>$400M 


~$11B 


*Source:  JRSG  Evaluation  of  Alternatives  Survey 


**Source:  Dan  Cuda,  Mike  Frieders,  IDA  CARD 
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JRSG  Systems  Engineering 
Objective  and  Constraints 

Objective:  Integrate  existing  Combined,  Joint,  Service,  and 
Agency  (C/J/S/A)  scenario  generation  capabilities  into  an 
enterprise  solution  that  can  rapidly  translate  authoritative 
data  into  a  set  of  initialization  products  that  support  mission 
critical  timelines. 

Constraints: 

•  Comply  with  Net-Centric  Data  Strategy  (NCDS)  and 
Universal  Core  (UC)  data  schema 

•  Utilize  Net-Centric  Enterprise  Services  (NCES) 

•  Synchronize  capability  development  with  Net-Enabled 
Combat  Capability  (NECC)  and  the  Command  and  Control 
(C2)  Domain  Core  data  schema 

•  Evolve  best  of  breed  C/J/S/A  capabilities 

•  Adhere  to  Information  Assurance  policy 


JRSG  Systems  Engineering  Approach 


Determine  JRSG  Demonstration  Objectives 
Integrate  existing  capabilities  as  enterprise  solution 
Determine  Combatant  Command  priority  data  sharing  needs 


Demonstrate  JRSG  Service  Oriented  Architecture  (SOA) 
Geospatial  Data  Discovery 


Map  Metadata  to  GSIP 


Build  Message  Broker 


Order  of  Battle  Data 
Discovery/Delivery 


Map  OOB  to  JC3IEDM 


JRSG  Community  of  Interest  (COI) 


Operations  Command 


Joint  Chiefs 
Of  Staff 


Forces 
Command 


Topographic 
Engineering  Center 


Program  Executive  Office 
Simulation  Training,  Instrumentation 


Synthetic 
Environment  Core 


DIS/^ 

Defense  Information  Systems  Agency 

Department  of  Defense 


National  Geospatial 
Intelligence  Agency 


US  Navy 


US  Air  Force 


US  Marine  Corps 


J  HI  F; 


Air  Force  Air  Force  Agency  Naval  Aviation 
Research  Lab  For  Modeling  System  Master  Plan 
&  Simulation 
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Notional  “As-ls”  Baseline  Capability 


Source  Data  Scenario  Build  Process  Target  Systems 


Conceptual  JRSG  “To  Be”  Architecture 

FOCUSED  ON  PROVIDING  LVC  FUTURE  IMMERSIVE  TRAINING  ENVIRONMENT 

Required  /  Potential  Data  Sources  1 


Force 
Structure 
DBs 


GIG 

Distribution 

Hosting 


NCES 

Discovery 
and  Delivery 

Collaboration 

Portal 

SOA 

Foundation 


JRSG  Services 


Define 

Event 


Schedule 

1 


Process 

Data 


User 

Obtain 

Release 

Management 

Data 

Product 

Common  Scenario  Definition 
Collaborative  Data  Workspace 
Data  Correlation 

Data  Configuration  Management 
Scenario  Data  Archive 
Translation  for  Export 


Mission 

Experiment 

Acquisition 

Test  & 

Training 

Mission 

Force 

Adaptive 

Mobility 

Decision 

Other 

Planning 

Apps 

Apps 

Apps 

Eval 

Apps 

Apps 

Rehearsal 

Apps 

Planning 

Apps 

Planning 

Apps 

Analysis 

Apps 

Support 

Apps 

Apps 

M&S  Enabled  Applications 


GIG:  Global  Information  Grid 
NCES:  Net-Centric  Enterprise  Services 
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JRSG  COI  Geospatial  Metadata  Mapping 


All  JRSG  COI  geospatial  discovery  metadata  mapped  to 
GEOINT  Structure  Implementation  Profile  (GSIP)  standard 

metadata  exchange  model. 


Order  of  Battle  Scenario  Generation  Data 


Across  Warfighter  and  Business  Domains 


ffS 


XSD  <* 
OUlDs 


Ops/Intel 

ADS 


Ops/lntel  Data  Linked  to  OUID 

- XSD’s  ~ 


^  **"  ^  XSD  m  Lqq 

-  \  OUlDs 

- L _ J  XSD’s 

ysd  —  OUlDs ""  » 


XSD  . 
'OUlDs 


XSD’s 


Joint 

Integrated 

Picture 


—  vcnv 

Pers  Data  Linked  to  OUID 


JRSG  SO  A 


Joint  Exercise 
Simulations 
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JRSG  SOA  Pilot  Operational  Nodes 


JFCOM  JTDS 


SOCOM  SOFPREP 


-) 

-) 


Army  TEC 


AFAMS/SGS 


o 


JCS  GFM  Dl 
Authorized 
Force 
Structure 


]/-K 

I  Federation  1 

T1 


1 

JRSG  SOA  I 

. 1 . 1 

JRSG  SOA 

Enterprise 

Enterprise 

Service 

Service 

Bus  (ESB)  i 

Bus  (ESB) 

Complex 

Complex 

Sub^PiSs 

Subscription 

Broker  (CSB) 

Broker  (CSB) 

SIPRNet 

NtPRNet 

L 

Cross  Domain 
Guard  -  TBD 

J 

H 

JFCOM  JTDS 

E- 

Navy  PSI 

M 

HI  Army  SE  Core 

H 

Air  Force/DMOC 

M 

Federation 

n 

EDCSS 
(ASNE- 
}  Weather) 

Web  Services 
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UNITED  STATES 


USJFCOM  -  IBM 

Cooperative  Research  and  Development  Agreement 

Joint  Force  Operations 
Service  Oriented  Architecture  (SOA) 

Applying  SOA 
9  October  2008 


Paul  Giangarra 
IBM  Distinguished  Engineer 
Office  of  the  CTO,  IBM  Federal 
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The  Path  to  Integrated  Systems 


Application  Application  Application 

1  2  3 


t  t  t 


Application  Application  Application 

1  2  3 


Silos 


Systems  of  Systems 


Integrated  Systems 
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What  is  (and  isn’t)  SOA? 


SOA  is... 


SOA  is  not.. 


Service  Oriented  Architecture 
A  way  of  thinking 

A  means  of  aligning  Business  with 
Information  Technology 

An  architectural  style  for  the 
design  of  business  applications  in 
terms  of  flexible,  reusable,  loosely 
coupled  service  assets 


•  A  standard 

•  A  specification 

•  A  programming  model 

•  A  platform 


The  SOA  Journey 


■joa 

fiidofi 


Build  Common 
Understanding  of  SOA 


PiJo< 
Work 

■yliVu' 

I 

Define  First  Project  and 
SOA  Roadmap 


Develop  Architecture  & 
Implementation  Road 
Map 


Demonstrate  SOA 

Production  Feasibility  Objective 


•  SOA  Overview 

•  SOA  Mission  Value 

•  SOA  Best  Practices 

•  SOA  Implementation 


SOA  Assessment 
Business  Process  Map 
SOA  Reference  Arch. 
SOA  Framework 
SOA  Governance 


Architecture  Principles 
Architecture  Decisions 
Integration  Framework 
Pilot  Project  Plan 


Create  Code 
Create  Documentation 
Establish  Governance 


Agenda 


•  Flandouts  •  Strategic  Roadmap 

•  Technical  Architecture 

•  IBIV 

1  SOA  Position  •  Project  Plan 

•  Definition  of  Phases 

•  SOA  Compass  Book  •  Service  Description 

•  Business  Case 

•  Mission  Value 

Production  ready  Code 

Technology  Proof 
Point 

Metrics 

Center  of  Excellence 


Deliverables 
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Joint  Rapid  Scenario  Generation  SOA  Reference 

Architecture 


Target  SYSTEMS 


OO 


JRSG  Processes 


Portal/User 

Interface 


Joint 

Organization 
Server  (GFM 
Dl) 


C 

0  </) 
£  ® 
Q..2 

°  t 

0  0 
>  co 
0  w 
Q 


Model 

Design 

Implement 

Test 


V 


V 


JRSG 
Common 
Databases 
(Geospatial, 
Weather,  etc) 


JFCOM  Data 
Catalog 


Business  Innovation  & 

Provide  for  bettei 
with  real-time  m  'n 


)ptimization  Services 

>r  iecision-making 
it  sion  information 


Interaction  Services 

Enables  collaboration 
between  people, 
processes  &  information 


Process  Services 

Orchestrate  and 
automate  business 
processes 


1 


Information  Services 

Manages  diverse  data 
and  content  in  a 
unified  manner 

ll 
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Enable  inter-connectivity  between  services 


Partner  Services 

Connect  with  mission 
partners 


Business  App  Services 

Access  Services 

42 

Build  on  a  robust, 
scaleable,  and  secure 
services  environment 

Facilitate  interactions 
with  existing  information 
and  application  assets 

Apps  & 

Info  Asst 

11  


Event 

Database 
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Information  Lifecycle:  The  “Problem”  Space 


Satellite 

Newsfeeds 

Radar 

UAV 


Complex  image  analysis 
Add  some  meta  data 
GPS  metadata,  target  analysis 
GPS  metadata 


Complex  Subscription 
Broker  fits  here 

Decouple  Ul  from  final 
information  “fusion”  and 
filtering 


Weather 


GPS  metadata 

■  ■  ■ 

Generically  steps: 

■Cleanse,  transform,  resolve,  combine  (federation), 
structure,  tag,  index 

■Choreograph  the  analysis  process 

■Requires  deterministic  E2E  responsiveness 


Community  based 
pub/sub 

Example  communities: 
jet  fighters,  bomber 
pilots,  AWACs,  AOC 
(various  roles) . 


16 


Key  Architectural  Decision 


Loosely  couple  the  “Front  End”  (Human  or  Machine 
Interface)  from  the  “Information  Broker”,  from  the  Information 
Source(s)  using  a  Complex  Subscription  Broker 


T7 


Use  Cases  to  Validate  Design  Assertions 

•  Publish  Terrain  Metadata 

•  Search  for  and  Request  Terrain  Data 

•  Receive  Terrain  Data 


(Sample)  Sequence  Diagrams  Created  to  document  the  use  cases: 
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To  Push  or  Pull:  Architectural  Alternatives 


TGSWeb 

Portal 


TGSWeb 

Portal 


External  Storage 


External 

Storage 


Metadata  Capture, 
Transform, 
Store, 


Source 

Provider 

System(s)^ 

Access  options:  FTP,  HTTP(S),  Web  Service 
Storage  Options:  File  system,  database 

Publish  Change  (Pub/Sub) 


Publish  Metadata  Capture, 
— .  Transform, 

UPdate  FI  Store, 


Email  Changes 


Email 

Update  \ 


Metadata  Capture, 
Transform, 
Store, 


Source 

Provider 

System(s) 

4  A 

Source 

Provider 

^System(s)^ 

Push  Options:  Source  Provider  pushes  the  metadata  outside  the  firewall(s^ 


Functional  Requirements 


Nonfunctional  Requirements 

(will  vary  by  site) 


Detect  change  in  Metadata 

Retrieve  updated  metadata  (can  be  entire 

catalog  at  first) 

Transform  local  metadata  structure  to  a 
common  format 

Store  common  format  metadata  (if  needed) 
Expose  metadata  to  search 


Requires  minimal  work  to  install,  configure 
and  manage  on  the  source  provider’s  part 
Adheres  to  IA  constraints 
IA:  Transparency  of  exposed  metadata 
IA:  Controlled  queries 
IA:  Controlled  ports/protocols 
Satisfy  agreed  to  SLAs 
Sources  are  geographically  distributed 
Source  provider  systems  may  not  always  be 
accessible 
Extensible  /  flexible 
Open  Architecture,  Standards  Based 


Controlled  Web  Service  Query 


Reverse  Proxy 


Agent 

Decomposed 


Pull  Options:  Source  responds  to  metadata  requests 


Logical  (Network  &  Product)  Architecture 


External 

Client 


Legend 

CSB  Plugin 
CSB  Agent  f 
Local 

Component 


Other 

Command 

Firewall 


Internal 

Client 

JTDS  Web  Pages 

Certificate 

Authority 
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Examples  of  What  is  Coming  Next 

•  Finalize  Security  Model  &  Design 

•  Finalize  the  Data  Model 

•  Design,  Develop,  Test  &  Deploy  the  Components 
and  Infrastructure 

•  Governance 

•  Possibly  Look  at  Alternative  Interface  Options 

•  Demonstrate  the  Results 

•  Determine  the  Next  Steps/Spirals 

•  Document  What  We  Learned 
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QUESTIONS????? 
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ESOH  Challenges  in 
Commissioning  an 
Aircraft  Carrier 


Doug  Parrish 
Booz  Allen  Hamilton 
11th  Systems  Engineering  Conference 


Summary 


•  Complex  operational  environment. 

•  Manning  challenges. 

•  Design/Contract  challenges. 

•  Equipment  challenges. 

•  ESOH  challenges. 

•  Hazardous  Materials 

•  Safety  Equipment 

•  Training 


USS  CONSTELLATION  (exCV-64) 


NGNN  Aircraft  Carriers 


Booz  Allen  Hamilton 
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Complex  Operations  I  Environment 

•  Busy  place.  NGNN  has  1000+  cranes, 
many  forklifts,  3  shifts  of  operation,  19k+ 
employees. 

-COMMERCIAL  YARD! 

•  Carrier  takes  5  years  to  build.  Some  crew 
there  ~2  years  prior  to  commissioning, 
phased  manning. 

•  Carrier  build  ~$5.5B  +  outfitting  + 
modernization.  ~  50M  manhours. 


NIMITZ  Class  (CVN-68) 

•  Builder:  Newport  News  Shipbuilding  Co,  NGNN/NGSB 

•  CVN-68  Deployed:  May  3, 1975. 

•  Unit  Cost:  ~  $4.5B  each,  +  planes  &  supplies. 

•  Propulsion:  2  nuclear  reactors,  4  shafts. 

•  Length:  1,092  ft 

•  Beam:  134  ft 

•  Flight  Deck  Width:  252ft 

•  Displacement:  ~  97k  tons  (88k  metric  tons)  full 
load. 

•  Speed:  30+  knots  (34.5+  mph). 

•  Crew:  Ship's  Company:  3,200  -  Air  Wing:  2,480. 

•  Aircraft:  85 


Booz  Allen  Hamilton 


KITTY  HAWK,  NIMITZ  AND  STENNIS- 
Intended  Area  of  Use  =  Complex  Operational 
Environment 


Design/Contract  Challenges 

•  1970s  Design. 

-Little  changed  from  first  NIMITZ  design. 

-Shipalts/mods  not  normally  done  at  yard, 
wait  on  PSA/SRA. 

-“As  designed/built”  to  pass  INSURV/Navy 
Acceptance  Trials,  then  many  items  ripped 
out/replaced  at  SRA. 

»Wet  Chemistry  Photolab. 

-FORD  design  -complete,  little  Fleet  input. 

»Too  late  to  input  ESOH  problems  now/not 
in  contract. 


Booz  Allen  Hamilton 
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BUSH,  2nd  with  new  bulbous  bow 


Photo:  Northrop  Grumman 


Booz  I  Allen  I  Hamilton 


BUSH  in  drvdock.  Mav06 


Superlifts:  Upper 


Bow,  Island 


3XQ 


BUSH  in  drvdock,  Sep06 
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PCU  BUSH  Christening-  Oct  7. 2006 


Booz  Allen  Hamilton 
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Manning  Challenges 

•  Few  people  initially.  Everyone  has 
multiple  jobs. 

•  As  Safety  Dept  and  rest  man  up,  most  are 
not  trained  for  primary  and  collateral 
duties. 

•  First  ship  or  carrier  tour  for  many. 

•  Safety  Dept  =  TAD  bodies. 


Booz  Allen  Hamilton 
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Schedule 


*  Keel  laid:  Sep  03 

*  First  crew  onboard:  Jun  06 

*  25% -Dec  06 

*  50% -Jun  07 

*  75% -Jan  08 

*  Light  off  reactor  Jul  08 

*  Crew  moveaboard  Aug  08 

*  Builder’s  Trials  Oct  08 

*  Navy  Acceptance  Trials  Dec  08 

*  Commissioning  Jan  09 

*  SRA/PSA  Mar  09 

*  Workups/FCT  late  09 

*  First  Cruise  late  2010 


Booz  Allen  Hamilton 


The  Ship 

•  BUSH  Contract  awarded  January  26,  2001. 

•  Seven-year  construction  timeframe. 

•  47,000  tons  of  structural  steel  and  about  a 
million  pounds  of  aluminum 

•  Modular  construction  process  forms  large 
individual  units  of  the  ship  much  like 
interlocking  building  blocks 

•  Units  welded  together  to  form  a  module  or 
superlift  weighing  up  to  900  tons. 


Booz  Allen  Hamilton 
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Booz  Allen  Hamilton 
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iftft 


Booz  Allen  Hamilton 
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Booz  Allen  Hamilton 
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Booz  I  Allen  I  Hamilton 
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The  Ship 


•  Top  speed  30+  knots. 

•  2  nuclear  reactors,  operate  20+  years  without 
refueling. 

•  ~50  years  lifespan. 

•  Three  two-inch  diameter  arresting  wires  on  the 
flight  deck  bring  an  airplane  going  150  MPH  to  a 
stop  in  <  400  ft. 


Booz  Allen  Hamilton 
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ESOH  Challenges 
During  Construction 


Equipment  Challenges 


•  Buy  initial  outfitting  items,  no  gear  comes 
with  the  job. 

•  AELs  are  wrong. 

•  Supply  Dept  undermanned,  no  HAZMAT 
program. 

•  RPPO  untrained. 

•  Byzantine  supply  system  (not  standard 
methodology). 


Booz  Allen  Hamilton 
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ESOH  Cha llenqes 

•  Getting  people  to  wear  PPE. 

•  Constant  training  challenges-  new  people, 
new  equipment,  new  systems,  complex 
operations. 

•  Commercial  yards  have  their  own  rules- 
some  are  arbitrary. 

•  Navy  DOES  NOT  OWN  THE  SHIP,  DOES 
NOT  OWN  THE  YARD. 

-Barge,  rented  offices,  Huntington  Hall. 


Booz  Allen  Hamilton 
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Street  Vie w 


Satellite 


Terrain 


ESOH  Challenges 


•  Shipyard  owns  emergency  (med,  spill, 
fire,  envm)  response  until  move-aboard. 

•  While  working  aboard,  follow  yard  rules- 
if  we  know/understand  them. 

•  SUPSHIP  is  intermediary. 


Booz  Allen  Hamilton 
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HAZMAT  Challenges 

•  One  BM2/9595  for  first  year  (Jul06-Dec07). 

•  No  AUL,  limited  visibility  on  ordering. 

•  SERVMART  provides  HAZMAT-  which 
may  be  fine  for  shore  offices  but  not 
usable  onboard. 

•  Safety  Dept  BM1/SK1  9595-  late  arrivals 
(Mar08). 

•  Have/use  HAZMAT  before  program  in 
place. 

-Training,  Hazcomm  standard,  PPE, 
disposal. 


Booz  Allen  Hamilton 
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Safety  Equipment  Not  Available 

Until  Crew  Moveaboard 


•  Just  Prior  To  Builder’s  Trials 

*  195  List/Exclusion  Items: 

-EEBDs  &  SCBAs. 

-Bull’s  Eyes,  CCOLs,  SIB. 

-Fire  fighting  equipment. 

-Ladder  chains. 

-Nonskid  decks. 

-Deck  coverings  &  deck  markings. 
-Warning  Labels/SOPS/Operator  Placards. 


Booz  Allen  Hamilton 


Training  Challenges 


•  Many  new,  unique,  untried  systems. 

•  Navy  crew  doesn’t  own  systems,  yard 
does  initially. 

•  Vendor  prepares  maintenance  +  training 
pubs-  often  late  in  the  game. 

•  Crew  must  be  trained/prepared  for  ATG 
Crew  Certification,  Builders  and  Navy 
Acceptance  Trials. 


Booz  Allen  Hamilton 
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Recommendations 


•  Standardize  and  implement  consistent, 
timely  SSWGs  and  allow  changes  to 
contract  and  design  early  in  design  cycle. 

•  More  fleet/user  community  involvement, 
earlier-  and  USE  their  suggestions. 

•  Make  and  use  passdown/lessons  learned. 

•  More  SUPSHIP  oversight  during  all 
phases  of  build  process. 


Booz  Allen  Hamilton 
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CVN-78  Plan 


Enhanced  Ship 
Self  Defense 


Improved  ffleaRQO 
&  Material  Handling 


UridijrvH^ilor 

Prcilecton 


Improved 

Survivability 


Zmnl  EJacirical 
DistfltkjUnn 
SysWn 


ProtMjJs-^sfi 

Plaints 


All  Ebdnt  Aim  Sarviesi 

New  PropiJlsion/Electric 

Plant 


Hnngftr  Buy*  (2) 


Image:  US  Navy 


^  EtetffcM&grauc;  Airmail 
LauncHing  Syslefns  [EMALS)  Mrcflft  E-kr^prs  (3) 
Jet  BI^fI  Ocflcctor? 

Enhanced  Flight  Deck 


Advanced 

Gm- 


MFRVER  P!  jrJar^, 


Join*  Plosion 
Approach  and  Landing 


E  barged 

Dee*: 

Footprint 

"PA 


Evriwd 

•Sen  Sp^rT'TVd 
missile 


OultHwd  Heavy  iJrtdtrway  RifpteniahrCHFfit 


DouKe  Heifltifl 


Integrated  Island 


Smaller  Island 

Composite  He-PosHionad 

Mf  5?  Aft  &  OultXKird 


Summary 

•  Complex  operational  environment. 

•  Manning  challenges. 

•  Design/Contract  challenges. 

•  Equipment  challenges. 

•  ESOH  challenges. 

•  Hazardous  Materials 

•  Safety  Equipment 

•  Training 


Booz  Allen  Hamilton 
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ESOH  Challenges  In 
ommissioning  an  Aircraft  Carrier 


Booz  Allen  Hamilton 
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Contact  info 


Douglas  K.  Parrish 

PhD,  CIH,  CSP,  REHS 

Booz  Allen  Hamilton 

Stafford  Commerce  Center,  Suite  103 

25  Center  Street 

Stafford,  VA  22556 

Phone  (540)  288-5126 

Fax  (540)  288-5050 

Parrish  Douqlas@bah.com 


Booz  Allen  Hamilton 
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Naval  Engineering  of  Systems 


I 


Translates 

Mss/on  Operational  Concepts  ->  Mission 

Capabilities 

Force  Focus 


1 


SoS 

Platform  /  Net  Centric 


Capability  Focus 

. , . , . t . . 


System 


* 

Functional  Focus 


t 


Component 

End  Item  Focus 

i 


Translates 
Mission  Capabilities  ->  System 
Requirements 


Sv 


Translates 
ystem  Requirements  ->  Component 
Functions 


Translates 

Component  Functions  ->  End  Items 
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Discussion  Topics 


IRda 

Chief 

Systems 

Engineer 


♦  Requiring  and  Acquiring  Alignment 

-  Program  Health 

♦  Net-Centric  Integration  and  Interoperability 

♦  System  Engineering  Processes 

♦  SE  Human  Resources 

♦  Software  Process  Improvement 


CHENG  Overview  to  TE  Forum  (8  Jan  07) 


DoN  Acquisition  Governance 


IRda 

Chief 
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♦  The  Secretary  of  the  Navy 

-  Comprehensive  review  of  the  Acquisition  process 

-  Challenges  in  Program  Planning  and  Execution. 

♦  Enhance  the  Acquisition  Governance  process 

-  Inject  Early  Senior  Leadership 

-  Continuous  Engagement  and  Transparency 

♦  Increase  discipline  during  each  phase  of  Program 
Maturity 

♦  Codified  by  SECNAVNOTE  on  26  February  2008 


“Two  Pass  /  Six  Gate” 
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DoN  Acquisition  Governance 


OSD/JOINT 

LEVEL 


JROC 


CSB 


NATY/USMC 

le\Ael 


FCD 

Approval 


Alternative 

Selection 


ASN(RD& A) 


ASN{RD&A) 


ASN(RD 

ASN(RD&A) 


Lead  Org:  OPNAV/HGMC 

Chair:  DCNQ  (NS}/DC,  CDSI 


OPNAV/HGMC 

CNQ/CMC 


OPNAV/HGMC 
CNO/CMC  - 


PEO/SYSCOM/ 

OPNAWHQMC 

LEVEL 


CONORS 

ODD 


SDS 

RFP 

Approval 

- - - ** 

Sufficiency  1 

.Approval 

V - r 

Review 

♦  First  Pass  -  Requirements  Establishment 


♦  Second  Pass  -  Acquisition  Execution 


♦  Gates  -  Reviews  to  Assess  Readiness  to  Proceed 


♦  System  Design  Specification  -  Capability  and  Performance 
Expectations 

CHENG  Overview  to  TE  Forum  (8  Jan  07) 


Naval 

Probability  of  Program  Success  (PoPS) 
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Naval  PoPS 


Program 

Requirements 

Program 

Resources 

Parameter  Status 

Budget 

Scope  Evolution 

Manning 

CON OPS 


Program 

Planning/Execution 


Industry /Company 
Assessment 


Technical  Maturity 


Cost  Estimating 


Sustainment 


Contract 

Execution 


Software 


External 

Influencers 


Acquisition 

Test  and 

Fit  in  DoD  & 

Management 

Evaluation 

Service  Visions 

Program  Advocacy 


Interdependencies 


Government  Program 
Office  P&rformantfi 
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Net-Centric  Integration  and  Interoperability  g 

Engineering  Management  systems 

^ 1^— ENGINEER 

♦  Transform  DoDAF  to  support  System  Engineering 

♦  Standard  Architecture  Data  Element  Reference  Guide 

♦  Jointly  issued  by  ASNRDA  and  DON  CIO 

♦  Naval  Enterprise  Architecture  Hierarchy 

♦  Approved  by  DON  CIO 

♦  Structured  Content  and  Format  to  retain  and  use  DODAF  Products 

♦  Manage  the  planning,  development,  testing,  and  fielding  of  Net- 
Centric  capabilities 

♦  Use  Information  Support  Plans  to  refine  System  and  Mission  evolutions. 

♦  Net  Ready  Key  Performance  Parameter  in  terms  that  can  be  Tested 

♦  Large  Scale  Capability  Evaluations  to  assess  System  and  Mission 
performance 


CHENG  Overview  to  TE  Forum  (8  Jan  07) 
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System  Engineering  Processes 


IRda 

Chief 

Systems 

Engineer 


“Lead  Systems  Integrator” 

-  Determine  the  Governments  role  at  the  Mission,  Net-Centric,  Platform, 
System,  and  Component  Levels 

Naval  SOS  Eng  Guidebook 

-  Issued  in  2006 

-  To  be  updated  to  better  support  Mission  Chief  Engineer  efforts 


System  Engineering  Technical  Review  (SETR)  Process 

-  ASNRDA  Policy  -  Execute  a  common  SETR  Process 

-  Ensure  Breadth  of  Technical  Functions  Infused 


Large  Scale  Capability  Evaluations 


Large  Scale  Capability  Evaluations 
Mission  -  SoS  -  System 


Rda 

© H1EF 

Systems 

/Engineer 


In  Service  Fleet  Exercises 


Prodixrts 

Acoui&rtion 
Gate  | 

Rcviovs  I - 


CD  <f> 

Pa»»  1 


TRA  SSR 


T  ecrin^cal 
Rev*ev,rs 


IRR 


ITR 


ASR 


SRR-I 

I 


SRR-II 

_ I 


SFR 

1 


PDR-I 

I 


IBR  PDR-II  CDR  TRR  FRR  OTRR  SVR/ 


PCA 


ISR 


FCA/PRR 


Experiments  /  DT  /  OT 


OPEVAL 


Post 

OPEVAL 


Mission  Threads 


Acquisition 
\<  iitrsrDne^' 

Phases 


Test  Scripts 


DODAF 


Mission  Architecture 


Near  /  Mid  /  Far  Term 
Capability  Assessments 
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SE  Human  Competency  Management 
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♦  Care  for  those  we  have 

-  Principal  DASN  for  Acquisition  Workforce 

-  On  Site  MS  in  System  Engineering  via  NPS  Embedded  Faculty 

•  NAVAIR  Pax  River 

•  NAVSEA  Dahlgren,  Port  Hueneme,  Newport,  and  Carderock 

-  Refine  KSA’s,  Education,  Training,  and  Job  Experiences 

♦  “Fill  the  Tub” 

-  Undergraduate  Candidates  through  Co-Opting,  Internships, 
Scholarships 

♦  “Prime  the  Pump” 

-  K-12  use  of  STEM 


CHENG  Overview  to  TE  Forum  (8  Jan  07) 
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Software  Process  Improvement 
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♦  ASNRDA  Issued  Software  Process  Improvement  Policy  and 
Guidebook 

-  Software  Acquisition  Management  (SAM) 

-  Software  Systems  Engineering  (SSE) 

-  Software  Development  Techniques  (SWDT) 

-  Business  Implications  (Bl) 

-  Human  Resources  (HR) 

♦  Software  Acquisition  Training  and  Education  Working 
Group  with  DAU,  OSD,  and  Services 

-  Program  Management  and  SPRDE  initial  focus 

♦  Quality,  Objective  Evidence  for  Assuring  SW 

-  Vulnerabilities,  Malicious  Code,  Security 


CHENG  Overview  to  TE  Forum  (8  Jan  07) 
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